Image processing method and device, computer device and storage medium

Abstract

This application provides an image processing method, apparatus, computer device, and storage medium, belonging to the field of computer technology. The method includes: acquiring a first image feature of a target image and a first text feature of target text, wherein the target image and the target text correspond to the same target object, and the target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part; performing a spatial transformation on the first image feature and the first text feature to obtain a second image feature and a second text feature, wherein the second image feature and the second text feature have the same dimension; fusing the second image feature and the second text feature to obtain a first target feature; and acquiring abnormal condition information of the target object based on the first target feature. The method provided by this application can improve the accuracy of acquiring abnormal condition information.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer device, and storage medium. Background Technology

[0002] In daily life, target objects may encounter some abnormal situations. When abnormal situations occur, it is necessary to predict the current abnormal situation information of the target object in a timely manner to avoid the abnormal situation causing a worse impact on the target object. The target object can refer to a variety of entities, such as equipment, human body, etc.

[0003] In related technologies, target images and target text of a target object are often acquired. The target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part. The image features of the target image and the text features of the target text are determined, and the image features and text features are concatenated. Based on the concatenated features, the abnormal condition information of the target object is obtained. However, because the dimensions of image features and text features are inconsistent, the accuracy of the abnormal condition information obtained in related technologies is poor. Summary of the Invention

[0004] This application provides an image processing method, apparatus, computer device, and storage medium, which can improve the accuracy of obtaining abnormal condition information of a target object. The technical solution is as follows:

[0005] On the one hand, an image processing method is provided, the method comprising:

[0006] The first image feature of the target image and the first text feature of the target text are obtained, wherein the target image and the target text correspond to the same target object, and the target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part;

[0007] The first image feature and the first text feature are spatially transformed to obtain the second image feature and the second text feature, which have the same dimension.

[0008] The second image features and the second text features are fused to obtain the first target features;

[0009] Based on the first target feature, obtain the abnormal status information of the target object.

[0010] On the other hand, an image processing method is provided, the method comprising:

[0011] Acquire a target image and target text, wherein the target image and the target text correspond to the same target object, and the target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part;

[0012] Based on the target image and the target text, an image processing model is invoked to obtain abnormal status information of the target object. The image processing model is used to perform spatial transformation on the first image feature of the target image and the first text feature of the target text to obtain a second image feature and a second text feature, wherein the second image feature and the second text feature have the same dimension; the second image feature and the second text feature are fused to obtain a first target feature; and based on the first target feature, abnormal status information of the target object is obtained.

[0013] On the other hand, an image processing apparatus is provided, the apparatus comprising:

[0014] The first acquisition module is used to acquire the first image features of the target image and the first text features of the target text, wherein the target image and the target text correspond to the same target object, and the target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part;

[0015] The transformation module is used to perform spatial transformation on the first image feature and the first text feature to obtain a second image feature and a second text feature, wherein the second image feature and the second text feature have the same dimension.

[0016] The fusion module is used to fuse the second image features and the second text features to obtain the first target features;

[0017] The second acquisition module is used to acquire abnormal status information of the target object based on the first target feature.

[0018] Optionally, the fusion module includes:

[0019] A first determining unit is configured to determine a first weight and a second weight, wherein the first weight corresponds to the second image feature and the second weight corresponds to the second text feature;

[0020] The fusion unit is used to perform weighted fusion of the second image features and the second text features based on the first weight and the second weight to obtain the first target features.

[0021] Optionally, the first determining unit is used to concatenate the first image features and the first text features to obtain the second target features; perform spatial transformation on the second target features to determine the first weight; and determine the second weight based on the first weight, wherein the sum of the first weight and the second weight is 1.

[0022] Optionally, the transformation module includes:

[0023] The second determining unit is used to determine the product of the first image feature and the first transformation coefficient to obtain the third image feature;

[0024] The third determining unit is used to determine the product of the first text feature and the second transformation coefficient to obtain the third text feature;

[0025] The processing unit is used to process the third image feature and the third text feature based on the activation function to obtain the second image feature and the second text feature.

[0026] Optionally, the device further includes:

[0027] The third acquisition module is used to acquire the target text;

[0028] The encoding module is used to encode the target text to obtain the first text feature.

[0029] Optionally, the third acquisition module includes:

[0030] The first acquisition unit is used to acquire the identity information of the target object;

[0031] The second acquisition unit is used to acquire at least one abnormality-related information of the target object, wherein the at least one abnormality-related information is used to reflect the abnormal condition of the target object;

[0032] The generation unit is used to generate the target text based on the identity information and at least one abnormality-related information.

[0033] On the other hand, an image processing apparatus is provided, the apparatus comprising:

[0034] The first acquisition module is used to acquire a target image and a target text, wherein the target image and the target text correspond to the same target object, and the target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part;

[0035] The calling module is used to call an image processing model based on the target image and the target text to obtain abnormal status information of the target object. The image processing model is used to perform spatial transformation on the first image feature of the target image and the first text feature of the target text to obtain a second image feature and a second text feature, the second image feature and the second text feature having the same dimension; the second image feature and the second text feature are fused to obtain a first target feature; and the abnormal status information of the target object is obtained based on the first target feature.

[0036] Optionally, the calling module includes:

[0037] A determining unit is used to determine the first text feature of the target text;

[0038] The processing unit is configured to input the target image and the first text feature into the image processing model and output the abnormal status information of the target object. The image processing model is also configured to determine the first image feature of the target image and receive the first text feature.

[0039] Optionally, the image processing model includes a network module, a fusion module, and a fully connected layer. The prediction unit is used to determine a first image feature of the target image through the network module; to perform spatial transformation on the first image feature and the first text feature through the fusion module to obtain a second image feature and a second text feature, wherein the second image feature and the second text feature have the same dimension; to fuse the second image feature and the second text feature to obtain a first target feature; and to obtain abnormal status information of the target object based on the first target feature through the fully connected layer.

[0040] Optionally, the device further includes:

[0041] The second acquisition module is used to acquire sample data, which includes sample images and sample target text. The sample images and sample target text correspond to the same sample object, and the sample data indicates the actual health status of the sample object.

[0042] The first determining module is used to determine the fourth text feature of the sample target text;

[0043] The second determining module is used to determine the fourth image feature of the sample image through the network module;

[0044] A fusion module is used to perform spatial transformation on the fourth image feature and the fourth text feature to obtain a fifth image feature and a fifth text feature, wherein the fifth image feature and the fifth text feature have the same dimension; and to fuse the fifth image feature and the fifth text feature to obtain a fourth target feature;

[0045] The third acquisition module is used to acquire abnormal status information of the target object based on the fourth target feature through the fully connected layer;

[0046] An update module is used to update the model parameters of the network module and the model parameters of the fusion module based on the acquired results and the actual results, respectively, to obtain the image processing model.

[0047] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the image processing method as described above.

[0048] On the other hand, a computer-readable storage medium is provided that stores at least one computer program, which is loaded and executed by a processor to perform the operations performed by the image processing method as described above.

[0049] On the other hand, a computer program product is provided, the computer program product comprising a computer program loaded and executed by a processor to perform the operations performed in the image processing method described above.

[0050] The image processing method, apparatus, computer equipment, and storage medium provided in this application embodiment convert the image features of the abnormal part of the target object and the text features of the target text into features of the same dimension through spatial transformation, so that the image features and text features have the same dimensions, thereby better fusing the image features and text features, and then obtaining abnormal situation information based on the fused features, which can improve the accuracy of obtaining abnormal situation information. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0053] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application;

[0054] Figure 3 This is a flowchart of an image processing method provided in an embodiment of this application;

[0055] Figure 4 This is a flowchart of an image processing method provided in an embodiment of this application;

[0056] Figure 5 This is a schematic diagram of a network structure for image processing provided in an embodiment of this application;

[0057] Figure 6 This is a flowchart of an image processing method provided in an embodiment of this application;

[0058] Figure 7 This is a flowchart of an image processing method provided in an embodiment of this application;

[0059] Figure 8 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0060] Figure 9 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0061] Figure 10 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0062] Figure 11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0063] Figure 12 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0064] Figure 13 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0066] It is understood that the terms "first," "second," etc., used in this application may be used to describe various concepts herein, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first image feature may be referred to as a second image feature, and similarly, a second image feature may be referred to as a first image feature.

[0067] "At least one" refers to one or more, for example, at least one piece of anomaly-related information can be one, two, three, or any integer greater than or equal to one. "Multiple" refers to two or more, for example, multiple pieces of anomaly-related information can be two, three, or any integer greater than or equal to two. "Each" refers to each of the at least one, for example, each piece of anomaly-related information refers to each of the multiple pieces of anomaly-related information. If the multiple pieces of anomaly-related information consist of three pieces of anomaly-related information, then each piece of anomaly-related information refers to each of the three pieces of anomaly-related information.

[0068] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0069] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0070] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0071] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0072] The image processing method provided in the embodiments of this application will be described below based on artificial intelligence technology and natural language processing technology.

[0073] The image processing method provided in this application embodiment can be used in a computer device. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smartphone, tablet computer, laptop computer, in-vehicle terminal, desktop computer, etc., but is not limited to these.

[0074] In one possible implementation, the computer program involved in the embodiments of this application may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network can form a blockchain system.

[0075] In one possible implementation, the computer device used to train the image processing model in this application embodiment is a node in a blockchain system. The node can store the trained image processing model in the blockchain. Then, the node or other nodes in the blockchain can call the image processing model to obtain abnormal status information of the target object based on the image of the abnormal part of the target object and the target text.

[0076] This application's embodiments are applied to computer devices. In the first scenario, the computer device includes a terminal, in which the terminal uses the image processing method provided in this application to obtain abnormal status information of the target object. In the second scenario, the computer device includes a server, in which the server uses the image processing method provided in this application to obtain abnormal status information of the target object. In the third scenario, the computer device includes both a terminal and a server, in which the terminal and the server jointly use the image processing method provided in this application to obtain abnormal status information of the target object. The following description uses a computer device including both a terminal and a server as an example.

[0077] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application. See also... Figure 1 The implementation environment includes a terminal 101 and a server 102. Terminal 101 can refer to one of multiple terminals; this embodiment uses terminal 101 as an example. Those skilled in the art will understand that the number of terminals can be more or less. Terminal 101 can run various types of applications, such as medical applications. Terminal 101 and server 102 are connected via a wireless or wired network, and terminal 101 can upload data to server 102 via the wireless or wired network. This data can be images and target text of abnormal parts of the target object collected by terminal 101.

[0078] Server 102 is used to execute an image processing method provided in this application embodiment. Server 102 and terminal 101 can be directly or indirectly connected via wired or wireless communication, which is not limited in this application embodiment. Optionally, the number of servers 102 can be more or less, which is not limited in this application embodiment. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services.

[0079] It should be noted that the images and target text of the abnormal parts of the target object processed in this application embodiment can be uploaded to the server 102 by the terminal 101 or obtained by the server 102 itself. This application embodiment does not limit this.

[0080] Another point to note is that in some embodiments, server 102 can use the image processing method provided in the embodiments of this application to obtain abnormal status information of the target object, or it can obtain abnormal status information of the target object through an image processing model; in other embodiments, based on the image processing service provided by server 102, terminal 101 can also use the image processing method provided in the embodiments of this application to obtain abnormal status information of the target object, or it can obtain abnormal status information of the target object through an image processing model.

[0081] After introducing the implementation environment of the embodiments of this application, the application scenarios of the embodiments of this application will be described below in conjunction with the above implementation environment. It should be noted that in the following description, the terminal is the terminal 101 mentioned above, and the server is the server 102 mentioned above.

[0082] According to the image processing method provided in this application embodiment, after terminal 101 uploads an image of an abnormal part of a target object and target text to server 102, server 102 obtains abnormal status information of the target object based on the image of the abnormal part of the target object and target text using the method provided in this application embodiment. Alternatively, it can obtain the abnormal status information of the target object through an image processing model, which can be applied to scenarios where abnormal status information of a target object is obtained. Based on the services provided by server 102, terminal 101 can also call the image processing model to obtain abnormal status information of the target object based on the image of an abnormal part of any target object and target text.

[0083] The image processing method provided in this application can be applied to any scenario where abnormal condition information of a target object is obtained.

[0084] In the first scenario, the target object is the human body. This image processing method can be applied to scenarios involving the acquisition of abnormal condition information. For example, the target object is the human body (patient), and the abnormal condition information could be lesions of internal organs. The target image is a CT (Computed Tomography) image of the human body, and the target text is medical record text. The computer device acquires the patient's CT image and medical record text, both of which are related to the patient's condition. By combining the CT image and medical record text, the computer device obtains the patient's condition information, predicts the patient's physical condition, and can predict the patient's current health status.

[0085] In the second scenario, where the target object is a device, this image processing method can be applied to obtaining information about the device's abnormal condition. For example, if the target object is a device (a component is damaged), the abnormal condition information could be the damage to a device component, the target image could be an image of the damaged component, and the target text could be text describing the device's damage. The computer device acquires both the target image and the target text related to the damage to the device component. By combining the target image and the target text, the computer device obtains information about the device's damage condition, predicts component damage, and can thus predict the device's current health status.

[0086] It should be noted that the embodiments in this application are only used as examples of human illness or damage to equipment components, and do not limit the application scenarios of the image processing method in this application.

[0087] Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The execution subject of this embodiment is a computer device; see [link to relevant documentation]. Figure 2 The method includes:

[0088] 201. The computer device acquires the first image features of the target image and the first text features of the target text.

[0089] In this context, the target image and target text correspond to the same target object, and the target image is an image of an abnormal part of the target object. This target image reflects the abnormality of the target object. For example, if the target object is a human body, then the target image is an image of the lesion in the abnormal part of the target object. For instance, the target image may be a CT image, an MRI (Magnetic Resonance Imaging) image, a CT image of an abnormal part of the target object after bone removal processing, or an MRI image of an abnormal part of the target object after bone removal processing; no specific limitation is made here. The computer device obtains the first image feature corresponding to the target image based on the target image. For example, the computer device obtains the first image feature through a convolutional neural network; no specific limitation is made here.

[0090] The target text records abnormal information related to the abnormal parts, and is text that reflects the abnormality of the target object. For example, if the target object is a human body (patient), then the target text is the target object's medical record text, which records information related to the target object's condition. The computer device obtains the corresponding first text feature based on the target text of the target object, and uses this first text feature to represent the target text information of the target object. For example, the computer device obtains the corresponding first text feature by inputting the abnormal information related to the target text into a word vector model; this is not specifically limited here.

[0091] 202. The computer device performs spatial transformation on the first image features and the first text features to obtain the second image features and the second text features.

[0092] The second image feature and the second text feature have the same dimension. In reality, images reflecting abnormal information of the target object are multi-dimensional images, meaning the acquired target image is a multi-dimensional image. Therefore, the first image feature corresponding to the target image is multi-dimensional, while the text feature of the target text is one-dimensional. That is, the dimensions of the first image feature and the first text feature are different.

[0093] In order to better integrate the image features of the target image and the text features of the target text, it is necessary to transform the image features of the target image and the text features of the target text to a feature space of the same dimension. Therefore, the computer device transforms the first image features and the first text features into second image features and second text features with the same dimension through spatial transformation.

[0094] 203. The computer device fuses the second image features and the second text features to obtain the first target features.

[0095] The first target feature is a fusion of the second image feature and the second text feature, which can reflect the abnormality of the target object from both the image of the abnormal part and the target text. The computer device fuses the image feature and the target text feature to obtain abnormal information related to the target object from multiple aspects, which is beneficial to the accuracy of subsequent acquisition of abnormality information of the target object.

[0096] 204. The computer equipment obtains information about the abnormal status of the target object based on the first target feature.

[0097] Computer equipment acquires information from both the image of the abnormal area and the target text based on the fused features. By combining this information, it obtains information about the abnormal condition of the target object and predicts its health status. The predicted health status can be categorized in several ways, such as "good health," "relatively good health," "relatively poor health," and "poor health." If the target object is a human body, the computer equipment can provide feedback to medical personnel, who can then develop a treatment plan. If the target object is equipment, the computer equipment can provide feedback to maintenance personnel, who can then develop a repair plan.

[0098] In this embodiment of the application, the computer device, based on the provided image processing method, transforms the image features of the abnormal part of the target object and the text features of the target text into features of the same dimension through spatial transformation, so that the image features and text features have the same dimensions, thereby better integrating the image features and text features, and then obtaining abnormal condition information of the target object based on the integrated features, which can improve the accuracy of obtaining abnormal condition information.

[0099] Figure 3 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The execution subject of this embodiment is a computer device; see [link to relevant documentation]. Figure 3 The method includes:

[0100] 301. Computer equipment acquires target images and target text.

[0101] In this context, the target image and target text correspond to the same target object. Before acquiring the target image and target text, the computer device needs to first determine the target object. For example, if the computer device determines the target object to be patient A, who has suffered a stroke, then the computer device acquires patient A's CT image and medical record text. The CT image reflects patient A's stroke condition, and the medical record text records relevant information related to patient A's stroke. Similarly, if the computer device determines the target object to be patient B, who has lung lesions, then the computer device acquires patient B's lung CT image and medical record text. The CT image reflects the condition of patient B's lung lesions, and the medical record text records relevant information related to patient B's lung lesions. Finally, if the computer device determines the target object to be a faulty piece of equipment, and a component of the equipment is damaged, then the computer device acquires an image of the damaged component and target text. The image reflects the extent of the damage, and the target text records abnormal information related to the damaged component.

[0102] 302. The computer device, based on the target image and target text, calls the image processing model to obtain abnormal status information of the target object.

[0103] The image processing model performs spatial transformation on the first image features of the target image and the first text features of the target text to obtain second image features and second text features, which have the same dimension. The second image features and second text features are then fused to obtain the first target feature. Based on the first target feature, abnormal status information of the target object is obtained. The computer device predicts the health status of the target object based on this abnormal status information, outputs the prediction result, and feeds the prediction result back to the user. The predicted health status can be categorized into several types, such as "good health," "relatively good health," "poor health," and "very poor health," etc.

[0104] In this embodiment, the computer device inputs the image of the abnormal part of the target object and the target text into the image processing model. The image processing model obtains image features and text features of the same dimension, so that the image features and text features are better fused. Based on the fused features, the abnormal situation information of the target object is obtained, which can improve the accuracy of obtaining abnormal situation information. Furthermore, by using the image processing model to obtain the abnormal situation information of the target object, the efficiency of obtaining abnormal situation information can be improved.

[0105] Figure 4 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The execution subject of this embodiment is a computer device. The example given uses a human body as the target object, a lesion image as the target image, and medical record text as the target text. See [link to relevant documentation]. Figure 4 The method includes:

[0106] 401. The computer device acquires the first image features of the target image lesion and the first text features of the medical record text.

[0107] Wherein, the target image and the target text correspond to the same target object, and the target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part, that is, the lesion image and the medical record text correspond to the same target object, and the lesion image is an image of a body part of the target object with a lesion, and the medical record text records information related to the condition of the lesion. The computer device obtains the first image feature of the target image and the first text feature of the medical record text, which can be accomplished through steps (1)-(2), including:

[0108] (1) The computer device acquires the first image features of the target image. This process can be completed in two steps, including:

[0109] (1-1) Computer equipment acquires target image.

[0110] In one possible implementation, the target image is a CT image of the lesion of the target object. The steps for the computer device to acquire the target image are: the computer device acquires the CT image of the lesion of the target object to obtain the target image. In this embodiment, the computer device directly uses the CT image of the lesion of the target object as the target image, which simplifies the operation and reduces processing power.

[0111] In another possible implementation, the target image is an MRI image of the lesion of the target object. The steps for the computer device to acquire the target image are: the computer device acquires the MRI image of the lesion of the target object to obtain the target image. In this embodiment, the computer device directly uses the MRI image of the lesion of the target object as the target image, which simplifies the operation and reduces processing power.

[0112] In another possible implementation, the target image is a boneless CT image of the lesion of the target object. The steps for the computer device to acquire the target image are: the computer device acquires the CT image of the lesion of the target object, and performs bone removal processing on the CT image to obtain the target image. In this embodiment, the computer device performs bone removal processing on the CT image, which reduces bone interference in subsequent image feature extraction. Based on this target image, the accuracy of acquiring abnormal condition information about the target object can be improved. Furthermore, based on this abnormal condition information, the accuracy of predicting the health status can be improved.

[0113] In another possible implementation, the target image is a de-osteoscopic MRI image of the lesion of the target object. The steps for the computer device to acquire the target image are: the computer device acquires the MRI image of the lesion of the target object, and performs de-osteoscopic processing on the MRI image to obtain the target image. In this embodiment, the computer device performs de-osteoscopic processing on the MRI image, which reduces bone interference in subsequent image feature extraction. Based on this target image, the accuracy of acquiring abnormal condition information about the target object can be improved. Furthermore, based on this abnormal condition information, the accuracy of predicting the health status can be improved.

[0114] (1-2) The computer device acquires the first image features based on the target image.

[0115] The computer device inputs the target image into a deep learning network, which then obtains the first image features of the target image. In this embodiment, the deep learning network can be a CNN (Convolutional Neural Network), DenseNet (Dense Network), or ResNet (Residual Network), etc., and is not specifically limited thereto.

[0116] (2) The computer device acquires the first textual features of the medical record text. This process can be completed in two steps, including:

[0117] (2-1) Computer equipment acquires target text (medical record text).

[0118] The target text records abnormal information related to the abnormal location, i.e., records disease-related information related to the lesion, including at least one abnormal information-related item (disease-related information) and identity information. Accordingly, the computer device acquires the target text in the following steps: the computer device acquires the identity information of the target object, acquires at least one abnormal information-related item of the target object, and generates the target text based on the identity information and at least one abnormal information-related item. The at least one abnormal information-related item is used to reflect the abnormal condition of the target object; in this embodiment, it is used to reflect the disease condition of the target object.

[0119] In one possible implementation, target objects of different ages and genders have different physical functions. Therefore, in the process of predicting the health status of the target objects, the computer device needs to obtain the target objects' age, gender and other identity information. Thus, the computer device obtains the target objects' identity information and uses this identity information as part of the target text. In this embodiment, the identity information may also include the target objects' name, ID number and other information. Based on this, the computer device can obtain the target text of the target objects. The form of the identity information is not specifically limited here.

[0120] In this embodiment of the application, the computer device obtains the target object's age, gender, and other identity information as target text information. Based on this information, abnormal condition information of the target object is obtained in subsequent processes, which can improve the accuracy of obtaining abnormal condition information. Furthermore, the health status of the target object can be predicted based on the abnormal condition information, thereby improving the accuracy of the prediction.

[0121] In another possible implementation, the abnormality-related information is a consciousness parameter, which reflects the target object's level of consciousness. Therefore, the computer device acquires the target object's consciousness parameter, for example, the Glasgow Coma Scale (GCS) score. Based on this score, the computer device can determine the target object's level of consciousness, i.e., whether the target object has a disturbance of consciousness. In this embodiment, the form of the consciousness parameter is not specifically limited.

[0122] In this embodiment of the application, the computer device acquires the consciousness parameters of the target object as target text information, thereby knowing the target object's state of consciousness. Based on this information, abnormal condition information of the target object can be acquired in subsequent processes, which can improve the accuracy of acquiring abnormal condition information. Furthermore, the target object's health status can be predicted based on the abnormal condition information, thereby improving the accuracy of the prediction.

[0123] In another possible implementation, the anomaly-related information is blood pressure parameters. The computer device acquires the blood pressure parameters of the target object, which represent the target object's blood pressure information. For example, these blood pressure parameters include systolic blood pressure and diastolic blood pressure. The systolic blood pressure parameter represents the blood pressure of the target object when the heart contracts, and the diastolic blood pressure parameter represents the blood pressure of the target object when the heart relaxes. The computer device acquires the target object's blood pressure information based on the systolic and diastolic blood pressure parameters.

[0124] In this embodiment of the application, the computer device acquires the systolic and diastolic blood pressure parameters of the target object as blood pressure parameters in the target text information, thereby improving the accuracy of knowing the target object's needs. Based on this information, abnormal condition information of the target object can be acquired in subsequent processes, which can improve the accuracy of acquiring abnormal condition information. Furthermore, based on the abnormal condition information, the health status of the target object can be predicted, thereby improving the accuracy of the prediction.

[0125] In another possible implementation, the abnormality-related information consists of multiple disease factors. Therefore, the computer device acquires multiple disease factors of the target object, where these factors are associated with lesions and indicate whether the target object has experienced symptoms corresponding to these disease factors. For example, if the target object has suffered a stroke, they may experience symptoms such as headache and vomiting. These symptoms reflect the target object's abnormality, and the computer device needs to acquire these symptoms to subsequently predict the target object's health status. Alternatively, multiple disease factors could be the target object's medical history information, such as a history of cerebral hemorrhage, cerebral infarction, hypertension, diabetes, hyperlipidemia, coronary heart disease, heart failure, or arrhythmia. These medical histories can all influence the target object's current health status.

[0126] In this embodiment of the application, the computer device acquires multiple disease factors of the target object as target text information. Based on this information, abnormal condition information of the target object is acquired in subsequent processes, which can improve the accuracy of acquiring abnormal condition information. Furthermore, the health status of the target object can be predicted based on the abnormal condition information, thereby improving the accuracy of the prediction.

[0127] The methods described above for obtaining abnormal information using computer devices can all be combined to obtain the target text, i.e., the computer generates the target text based on identity information, consciousness parameters, blood pressure parameters, and multiple disease factors; or, some of the above methods can be combined to obtain the target text, i.e., the computer generates the target text based on identity information, consciousness parameters, and blood pressure parameters, or the computer generates the target text based on identity information, consciousness parameters, and multiple disease factors. No specific limitations are made here.

[0128] It should be noted that computer devices can also acquire annotation information to generate target text. For example, this annotation information could include the target's lifestyle habits, such as whether the target smokes, drinks alcohol, etc. While not specifically limited here, these lifestyle habits can affect the target's current health status, and the computer device needs to acquire this information. For instance, the annotation information could also include whether the target is unconscious, whether there has been bleeding into the ventricles, whether anticoagulation therapy is being administered, or whether antiplatelet therapy is being administered. This information can reflect any abnormal conditions of the target.

[0129] In this embodiment of the application, the computer device acquires multiple abnormal information related to the target object, and constructs target text from these multiple abnormal information related to the target object. In subsequent processes, the abnormal status information of the target object can be obtained through the target text containing multiple abnormal information related to the target object, which can improve the accuracy of obtaining the abnormal status information. Furthermore, the health status of the target object can be predicted based on the abnormal status information, thereby improving the accuracy of the prediction.

[0130] (2-2) The computer device encodes the target text to obtain the first text feature.

[0131] Computer equipment encodes multiple pieces of information in the target text, such as identity information, consciousness parameters, blood pressure parameters, and multiple disease factors, to obtain the first text feature. The following explanation uses a one-hot encoding method for encoding the target text as an example. The computer first classifies the multiple pieces of information in the target text, such as identity information, consciousness parameters, blood pressure parameters, and multiple disease factors, and then performs one-hot encoding on the classification results.

[0132] Regarding age in identity information, computer devices can classify age into multiple categories based on its numerical value. For example, age can be divided into seven categories: under 30 years old, 30-39 years old, 40-49 years old, 50-50 years old, 60-69 years old, 70-79 years old, and not less than 80 years old. Computer devices use 8-bit encoding for the age of the target object.

[0133] For the consciousness parameters of the target object, the computer device can classify the consciousness parameters into multiple categories according to their values. For example, the consciousness parameter is the GCS score, which is divided into four levels: 15 points indicates that the target object is normal, 13-14 points indicates that the target object has mild consciousness impairment, 9-12 points indicates that the target object has moderate consciousness impairment, and 2-8 points indicates that the target object has severe consciousness impairment. The computer device uses 5-bit encoding for the consciousness parameters of the target object.

[0134] For the target subject's blood pressure parameters, which include systolic and diastolic blood pressure, the computer equipment can classify the systolic and diastolic blood pressure parameters according to their values. For example, based on the magnitude of the systolic blood pressure parameter, the computer equipment can classify it into three categories: the first category is a systolic blood pressure parameter below 90, indicating that the target subject has low blood pressure; the second category is a systolic blood pressure parameter above 140, indicating that the target subject has high blood pressure; and the third category is a systolic blood pressure parameter between 90 and 140, indicating that the target subject's blood pressure is normal. The computer equipment can also classify the diastolic blood pressure parameter into three categories: the first category is a diastolic blood pressure parameter below 60, indicating that the target subject has low blood pressure; the second category is a diastolic blood pressure parameter above 90, indicating that the target subject has high blood pressure; and the third category is a diastolic blood pressure parameter between 60 and 90, indicating that the target subject's blood pressure is normal. The computer equipment uses 4-bit encoding for both the systolic and diastolic blood pressure parameters of the target subject.

[0135] Information such as gender, multiple disease factors, whether the person is in a coma, whether there is bleeding into the ventricle, whether anticoagulation therapy is being administered, and whether antiplatelet therapy is being administered can all be divided into two categories. Computer equipment uses 3-bit encoding for this information.

[0136] One point to note is that for each piece of information, a default class (information is empty) is added. Taking age as an example, the computer device divides age into 7 categories, so age is encoded using 8 bits. Taking GCS score as an example, the computer device divides GCS score into 4 categories, so GCS score is encoded using 5 bits. Taking history of hypertension as an example, the computer device divides history of hypertension into two categories: "yes" and "no". The "yes" category indicates that the target subject's medical history includes a history of hypertension, and the "no" category indicates that the target subject does not have a history of hypertension, so history of hypertension is encoded using 3 bits.

[0137] For example, the computer retrieves 21 pieces of information, including age, gender, Glasgow Coma Scale (GCS) score, headache, vomiting, coma, whether bleeding has entered the ventricles, history of cerebral hemorrhage, history of cerebral infarction, history of hypertension, history of diabetes, history of hyperlipidemia, history of coronary heart disease, history of heart failure, history of arrhythmia, anticoagulation, antiplatelet therapy, smoking, alcohol consumption, admission systolic blood pressure, and admission diastolic blood pressure. Age is encoded using 8 bits, GCS score using 5 bits, diastolic and systolic blood pressure using 4 bits, and other information using 3 bits. The final case information encoding vector dimension = 8 + 5 + 4 + 4 + 3 * 17 = 72.

[0138] 402. The computer device performs a spatial transformation on the first image feature and the first text feature to obtain the second image feature and the second text feature.

[0139] The computer device performs spatial transformations on the first image features and the first text features respectively to obtain corresponding second image features and second text features. The spatial transformation can be implemented using a fully connected network, which is not specifically limited here. The process by which the computer device performs spatial transformations on the first image features and the first text features to obtain the second image features and second text features can be implemented through the following steps:

[0140] (1) The computer device determines the product of the first image feature and the first transformation coefficient to obtain the third image feature.

[0141] Wherein, the first transformation coefficients are the coefficients used by the computer device to perform spatial transformation on the first image features. For example, see... Figure 5 Wherein input1 is the first image feature, W1 is the first transformation coefficient, the computer device transforms the first image feature input1 through W1, and multiplies the first transformation coefficient W1 with the first image feature input1 to obtain the third image feature.

[0142] (2) The computer device determines the product of the first text feature and the second transformation coefficient to obtain the third text feature.

[0143] The second transformation coefficients are the coefficients used by the computer device to spatially transform the first text features. For example, see [link to previous section]. Figure 5 Wherein input2 is the first text feature, W2 is the second transformation coefficient, the computer device transforms the first text feature input1 through W2, and multiplies the second transformation coefficient W2 with the first text feature input2 to obtain the third text feature.

[0144] (3) The computer device processes the third image feature and the third text feature based on the activation function to obtain the second image feature and the second text feature.

[0145] The activation function can be either the Tanh (hyperbolic tangent) function or the sigmoid function; no specific limitation is made here. For example, see [link to previous section]. Figure 5 The computer device uses the Tanh function as the activation function to process the third image features, and obtains the second image features using the following formula:

[0146] Formula 1:

[0147] h1 = tanh(W1*x1)

[0148] Where h1 is the feature vector of the second image feature, used to represent the second image feature, tanh is the hyperbolic tangent function, W1*x1 is used to represent the third image feature, W1 is the first transformation coefficient, and x1 is the feature vector of the first image feature, used to represent the first image feature.

[0149] The computer device uses the Tanh function as the activation function to process the third text feature, and obtains the second text feature using the following formula:

[0150] Formula 2:

[0151] h2 = tanh(W2*x2)

[0152] Where h2 is the feature vector of the second text feature, used to represent the second text feature, tanh is the hyperbolic tangent function, W2*x2 is used to represent the third text feature, W2 is the second transformation coefficient, and x2 is the feature vector of the first text feature, used to represent the first text feature.

[0153] 403. The computer equipment determines the first and second weights.

[0154] Wherein, the first weight corresponds to the second image feature, and the second weight corresponds to the second text feature. The computer device can determine the first weight and the second weight using the first image feature and the second image feature. Accordingly, the steps for the computer device to determine the first weight and the second weight are as follows: the computer device concatenates the first image feature and the first text feature to obtain the second target feature; performs a spatial transformation on the second target feature to determine the first weight; and determines the second weight based on the first weight, wherein the sum of the first weight and the second weight is 1.

[0155] For example, see continue. Figure 5 The computer device concatenates the first image feature input1 and the second image feature input2 along the vector dimension, performs a spatial transformation on the concatenated features, and determines the first weight using the following formula three:

[0156] Formula 3:

[0157] z = sigmod(Wz*[x1,x2])

[0158] Where z is the first weight, which is the weight of the second image feature; the sigmoid function is the activation function; Wz is the transformation coefficient for spatial transformation of the second target feature; [x1, x2] is the feature vector of the second target feature, used to represent the second target feature; x1 is the feature vector of the first image feature, used to represent the first image feature; and x2 is the feature vector of the first text feature, used to represent the first text feature. Since the first weight is z, and the sum of the first and second weights is 1, the second weight is 1-z.

[0159] 404. The computer device performs weighted fusion of the second image features and the second text features based on the first weight and the second weight to obtain the first target feature.

[0160] The first weight determines the contribution of the first image feature to the fused first target feature, and the second weight determines the contribution of the first text feature to the fused first target feature.

[0161] For example, see continue. Figure 5 The computer device, based on the first and second weights, performs a weighted fusion of the second image features and the second text features using the following formula four to obtain the first target feature:

[0162] Formula 4:

[0163] x_out = z*h1 + (1-z)*h2

[0164] Where x_out is the feature vector of the first target feature, used to represent the first target feature, z is the first weight, h1 is the feature vector of the second image feature, used to represent the second image feature, (1-z) is the second weight, and h2 is the feature vector of the second text feature, used to represent the second text feature.

[0165] It should be noted that after the computer device processes the features through spatial transformation, the resulting features are identical in dimension. That is, the second image feature, the second text feature, and the first target feature have the same dimension. For example, through spatial transformation, the dimensions of the resulting second image feature, second text feature, and first target feature are all kept at 48.

[0166] 405. The computer equipment obtains information about the abnormal status of the target object based on the first target feature.

[0167] The computer device acquires information from both the image of the abnormal area and the target text based on the fused features. By combining these two pieces of information, it obtains information about the abnormal condition of the target object, predicts its health status based on this information, and outputs the prediction result, which is then fed back to the user. The predicted health status can be categorized in several ways, such as "good health," "relatively good health," "relatively poor health," and "poor health." The following example illustrates how the computer device uses the GOS (Glasgow Outcome Scale) score to predict the health status of a target object:

[0168] The GOS score is divided into five levels: a GOS score of 1 indicates that the target subject is dead; a GOS score of 2 indicates that the target subject is vegetative and has minimal response (e.g., the eyes can open during sleep / wake cycles); a GOS score of 3 indicates that the target subject is severely disabled, conscious, and requires daily care; a GOS score of 4 indicates that the target subject is mildly disabled but can live independently and work under protection; and a GOS score of 5 indicates that the target subject has recovered well and returned to a normal life, despite having mild defects.

[0169] Each target image has a corresponding GOS score, and each first target feature corresponds to a GOS score. The computer device can divide the first target features into two categories based on the GOS score. For example, the computer device classifies the first target features with a GOS score of no more than 3 into one category, and the prediction result for this category is poor health status. The computer device classifies the first target features with a GOS score of more than 3 into another category, and the prediction result for this category is good health status.

[0170] In this embodiment of the application, the computer device, based on the provided image processing method, transforms the image features of the abnormal part of the target object and the text features of the target text into features of the same dimension through spatial transformation, so that the image features and text features have the same dimensions. Furthermore, the image features and text features with the same dimensions are weighted and fused, which improves the fusion effect. Then, based on the fused features, the abnormal condition information of the target object can be obtained, which can improve the accuracy of obtaining abnormal condition information.

[0171] Figure 6 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The execution subject of this embodiment is a computer device, and the description uses the target object as an example of the device. See [link to documentation]. Figure 6 The method includes:

[0172] 601. The computer device acquires the first image features of the target image and the first text features of the target text.

[0173] The target image is an image of the abnormal part of the target object, and the target text records abnormal information related to the abnormal part. That is, the target image is an image of the damaged part of the equipment, and the target text records abnormal information related to the damaged part. Step 601 can be completed through steps (1)-(2), including:

[0174] (1) The computer device acquires the first image features of the target image.

[0175] The method for obtaining the first image feature of the target image in this step is the same as the method for obtaining the first image feature of the target image in step 401, and will not be described again here.

[0176] (2) The computer device acquires the first textual features of the medical record text. This process can be completed in two steps, including:

[0177] (2-1) Computer equipment acquires target text.

[0178] The target text records abnormal information related to the abnormal parts, specifically, it records abnormal information related to the damaged parts. This includes at least one abnormal information and identity information. Accordingly, the computer device acquires the target text by: acquiring the identity information of the target object; acquiring at least one abnormal information related to the target object; and generating the target text based on the identity information and at least one abnormal information related to the target object. The at least one abnormal information related to the target object reflects its abnormal condition; in this embodiment, it reflects the damage condition of the target object.

[0179] In one possible implementation, target objects of different brands and different types have different performance characteristics. Therefore, in the process of predicting the health status of the target object, the computer device needs to obtain the target object's brand, model, and other identity information. Thus, the computer device obtains the target object's identity information and uses this identity information as part of the target text. In this embodiment, the identity information may also include the target object's production time, production batch number, and other information. Based on this, the computer device can obtain the target text of the target object. The form of the identity information is not specifically limited here.

[0180] In this embodiment of the application, the computer device obtains the brand, type and other identity information of the target object as target text information. Based on this information, abnormal status information of the target object is obtained in the subsequent process, which can improve the accuracy of obtaining abnormal status information. Then, the health status of the target object can be predicted based on the abnormal status information, which can improve the accuracy of prediction.

[0181] In another possible implementation, the abnormal information is a wear parameter, which is used to reflect the wear condition of the target object. The wear can be a phenomenon that changes in size, shape or surface quality of the equipment due to friction or vibration during use. For example, the wear can be caused by friction between the parts of the equipment during long-term use, or it can be caused by human factors such as handling or using the equipment. No specific limitation is made here.

[0182] In this embodiment of the application, the computer device acquires the wear parameters of the target object as target text information, thereby knowing the wear status of the target object. Based on this information, abnormal condition information of the target object can be acquired in subsequent processes, which can improve the accuracy of acquiring abnormal condition information. Furthermore, the health status of the target object can be predicted based on the abnormal condition information, thereby improving the accuracy of the prediction.

[0183] In another possible implementation, the anomaly-related information is a corrosion parameter, which is used to reflect the corrosion status of the target object. This corrosion can be caused by chemical factors affecting the equipment under natural conditions. For example, if there is metal in the equipment, the metal may react chemically directly with the surrounding medium (e.g., oxygen), or the metal may react electrochemically with a dielectric solution (e.g., water). These will all produce corrosion, and no specific limitation is made here.

[0184] In this embodiment of the application, the computer device obtains the corrosion parameters of the target object as target text information, thereby knowing the corrosion status of the target object. Based on this information, abnormal status information of the target object can be obtained in subsequent processes, which can improve the accuracy of obtaining abnormal status information. Furthermore, the health status of the target object can be predicted based on the abnormal status information, thereby improving the accuracy of the prediction.

[0185] In another possible implementation, the abnormal information is a fracture parameter, which reflects the fracture condition of the target object. The fracture may be caused by improper processing of the equipment, or by fatigue caused by the equipment due to long-term operation (e.g., high temperature fatigue, mechanical fatigue), or by human factors during handling or use. No specific limitation is made here.

[0186] In this embodiment of the application, the computer device obtains the fracture parameters of the target object as target text information, thereby knowing the fracture status of the target object. Based on this information, abnormal status information of the target object can be obtained in subsequent processes, which can improve the accuracy of obtaining abnormal status information. Furthermore, the health status of the target object can be predicted based on the abnormal status information, thereby improving the accuracy of the prediction.

[0187] In another possible implementation, the anomaly-related information is the working time, which reflects the time from the date of production to the current working time of the target object. During use, the quality of equipment continuously declines, and each piece of equipment has its own warranty period. If the working time of equipment exceeds its warranty period, the equipment is prone to failure. Therefore, when obtaining anomaly information of a target object, the computer needs to obtain the target object's working time as part of the target text.

[0188] In this embodiment of the application, the computer device obtains the working time of the target object as target text information. Based on this information, abnormal status information of the target object is obtained in subsequent processes, which can improve the accuracy of obtaining abnormal status information. Furthermore, the health status of the target object can be predicted based on the abnormal status information, thereby improving the accuracy of the prediction.

[0189] The methods described above for obtaining abnormal information from computer devices can all be combined to obtain the target text, i.e., the computer generates the target text based on identity information, wear parameters, corrosion parameters, fracture parameters, and operating time; or, some of the above methods can be combined to obtain the target text, i.e., the computer generates the target text based on identity information, wear parameters, and fracture parameters, or the computer generates the target text based on identity information, operating time, and corrosion parameters. No specific limitations are made here.

[0190] In this embodiment of the application, the computer device acquires multiple abnormal information related to the target object, and constructs target text from these multiple abnormal information related to the target object. In subsequent processes, the abnormal status information of the target object can be obtained through the target text containing multiple abnormal information related to the target object, which can improve the accuracy of obtaining the abnormal status information. Furthermore, the health status of the target object can be predicted based on the abnormal status information, thereby improving the accuracy of the prediction.

[0191] (2-2) The computer device encodes the target text to obtain the first text feature.

[0192] The method for encoding the target text in this step is the same as the method for encoding the target text in step 401, and will not be repeated here.

[0193] 602. The computer device performs a spatial transformation on the first image feature and the first text feature to obtain the second image feature and the second text feature.

[0194] 603. The computer equipment determines the first and second weights.

[0195] 604. The computer device performs weighted fusion of the second image features and the second text features based on the first weight and the second weight to obtain the first target feature.

[0196] Steps 602-604 are the same as steps 402-404, and will not be repeated here.

[0197] 605. The computer equipment obtains information about the abnormal status of the target object based on the first target feature.

[0198] The computer device acquires information from two aspects: the image of the abnormal area and the target text, based on the fused features. By combining these two pieces of information, it obtains information about the abnormal condition of the target object. Based on this abnormal condition information, it predicts the health status of the target object and outputs the prediction result, which is then fed back to the user. This prediction result can be divided into two types: good health status and poor health status. The following example illustrates how the computer device uses equipment aging scoring to predict the health status of the target object:

[0199] The aging score is divided into five levels: an aging score of 1 indicates that the target object is severely aged, meaning that the equipment is unusable; an aging score of 2 indicates that the target object is relatively severely aged, meaning that the equipment can only be put into use after the aging parts are replaced; an aging score of 3 indicates that the target object is moderately aged, meaning that the equipment can only be put into use after maintenance; an aging score of 4 indicates that the target object is slightly aged, meaning that the equipment can be put into use after simple maintenance; and an aging score of 5 indicates that the target object is slightly aged, meaning that the equipment has minor defects, but does not affect its use.

[0200] Each target image has a corresponding aging score, and each first target feature corresponds to an aging score. The computer device can divide the first target features into two categories based on the aging score. For example, the computer device classifies the first target features with an aging score of no more than 3 into one category, and the prediction result for this category is poor health status. The computer device classifies the first target features with an aging score of more than 3 into another category, and the prediction result for this category is good health status.

[0201] In this embodiment of the application, the computer device, based on the provided image processing method, transforms the image features of the abnormal part of the target object and the text features of the target text into features of the same dimension through spatial transformation, so that the image features and text features have the same dimensions. Furthermore, the image features and text features with the same dimensions are weighted and fused, which improves the fusion effect. Then, based on the fused features, the abnormal condition information of the target object can be obtained, which can improve the accuracy of obtaining abnormal condition information.

[0202] In the above Figure 4Based on the provided image processing method, the computer device can also train an image processing model that can acquire abnormal status information of the target object. The computer device calls the trained image processing model to acquire abnormal status information of the target object. For details, please refer to the following embodiments.

[0203] Figure 7 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The execution subject of this embodiment is a computer device; see [link to relevant documentation]. Figure 7 The method includes:

[0204] 701. Computer equipment acquires target images and target text.

[0205] In this context, the target image and target text correspond to the same target object, with the target image being an image of an abnormal part of the target object, and the target text recording abnormal information related to the abnormal part. The method by which the computer device acquires the target image and target text is the same as the method in step 401, and will not be repeated here.

[0206] 702. The computer equipment determines the first text features of the target text.

[0207] The process by which the computer device determines the first text feature of the target text in this step is the same as the process of obtaining the first text feature of the target text in step 401, and will not be described again here.

[0208] 703. The computer equipment inputs the target image and the first text features into the image processing model to obtain information on the abnormal status of the target object.

[0209] The image processing model is used to perform spatial transformation on the first image features of the target image and the first text features of the target text to obtain second image features and second text features, which have the same dimension; the second image features and second text features are fused to obtain the first target features; based on the first target features, abnormal status information of the target object is obtained; and the image processing model is also used to determine the first image features of the target image and receive the first text features.

[0210] The image processing model includes a network module, a fusion module, and a fully connected layer. The process by which a computer device calls the image processing model to obtain abnormal status information of the target object can be implemented through steps 7031-7033, including:

[0211] 7031. The computer device determines the first image feature of the target image through the network module.

[0212] The process by which the computer device determines the first image feature in this step is the same as the process of acquiring the first image feature in step 401, and will not be described again here.

[0213] 7032. The computer device performs spatial transformation on the first image feature and the first text feature through a fusion module to obtain the second image feature and the second text feature, the second image feature and the second text feature having the same dimension; the second image feature and the second text feature are fused to obtain the first target feature.

[0214] 7033. The computer device obtains abnormal status information of the target object based on the first target feature through the fully connected layer.

[0215] The process of determining the abnormal status information of the target object in steps 7032-7033 is the same as the process of obtaining the abnormal status information of the target object in steps 402-405, and will not be described again here.

[0216] For example, an image processing model consists of a DenseNet-121 network and a GMU (Gated Multimodal Unit). That is, a GMU fusion unit is added before the fully connected layer of the DenseNet-121 network. The computer device calls this image processing model to obtain abnormal status information of the target object.

[0217] It should be noted that the computer device can also directly input the target image and target text into the image processing model to obtain the abnormal status information of the target object. Accordingly, the steps for the computer device to call the image processing model based on the target image and target text to obtain the abnormal status information of the target object are as follows: the computer device inputs the target image and target text into the image processing model and outputs the abnormal status information of the target object. The image processing model is also used to determine the first image feature of the target image and the first text feature of the target text.

[0218] Another point to note is that the computer device can compare the output of the image processing model with the actual results, and update the model parameters of the network module and the fusion module in the model. Accordingly, the process of updating the model parameters of the network module and the fusion module in the model is achieved through the following steps:

[0219] (1) Computer equipment acquires sample data, which includes sample images and sample target text. The sample images and sample target text correspond to the same sample object, and the sample data labels the actual results of the health status of the sample object.

[0220] (2) Computer equipment determines the fourth text feature of the target text of the sample;

[0221] (3) The computer equipment determines the fourth image feature of the sample image through the network module;

[0222] (4) The computer device performs spatial transformation on the fourth image feature and the fourth text feature through the fusion module to obtain the fifth image feature and the fifth text feature, which have the same dimension; the fifth image feature and the fifth text feature are fused to obtain the fourth target feature;

[0223] (5) The computer device obtains abnormal status information of the target object based on the fourth target feature through the fully connected layer;

[0224] The process of the computer device obtaining abnormal status information of the target object in steps (1)-(5) and the process of obtaining abnormal status information of the target object in steps 701-703 will not be described again here.

[0225] (6) Based on the acquired results and the actual results, the computer equipment updates the model parameters of the network module and the model parameters of the fusion module respectively to obtain the image processing model.

[0226] The network module's model parameters determine the image features of the sample images, while the fusion module's model parameters fuse the image and text features of the sample data. The computer trains this image processing model using the sample data. During training, the model parameters are continuously updated by comparing the obtained results with actual results, thereby improving the accuracy of the image processing model in acquiring information about abnormal situations.

[0227] For example, if the target object is the human body, the computer equipment uses 1990 CT images containing GOS scores and medical record information as sample data to train the health training model. Specifically, the computer equipment determines the sample data with a GOS score of no more than 3 as positive samples and the sample data with a GOS score of less than 3 as negative samples. The 1990 sample data are randomly divided into training set, validation set, and test set, and the ratio of training set:validation set:test set is 6:2:2. During training, the batch size is set to 8, the learning rate is set to 1e-4, and four NVIDIA Tesla V100 graphics cards are used for training.

[0228] In this embodiment, the computer device inputs an image of the abnormal part of the target object and the target text into an image processing model. The image processing model acquires image features and text features of the same dimension, enabling better fusion of image features and text features. Based on the fused features, abnormal condition information of the target object is obtained, which can improve the accuracy of obtaining abnormal condition information. Furthermore, obtaining abnormal condition information of the target object through this image processing model can improve the efficiency of obtaining abnormal condition information. At the same time, the computer device uses a model trained with sample data, which can improve the accuracy of obtaining abnormal condition information.

[0229] Figure 8 This is a flowchart of an image processing apparatus provided in an embodiment of this application. See also... Figure 8 The device includes:

[0230] The first acquisition module 801 is used to acquire the first image features of the target image and the first text features of the target text. The target image and the target text correspond to the same target object, and the target image is an image of an abnormal part of the target object. The target text records abnormal information related to the abnormal part.

[0231] Transformation module 802 is used to perform spatial transformation on the first image features and the first text features to obtain second image features and second text features, wherein the second image features and the second text features have the same dimension.

[0232] The fusion module 803 is used to fuse the second image features and the second text features to obtain the first target features;

[0233] The second acquisition module 804 is used to acquire abnormal status information of the target object based on the first target feature.

[0234] The health status prediction device provided in this application transforms the image features of the abnormal part of the target object and the text features of the target text into features of the same dimension through spatial transformation, so that the image features and text features have the same dimensions, thereby better integrating the image features and text features, and then predicting the health status based on the integrated features, which can improve the accuracy of the prediction.

[0235] Optionally, see Figure 9 The fusion module 803 includes:

[0236] The first determining unit 813 is used to determine a first weight and a second weight, wherein the first weight corresponds to a second image feature and the second weight corresponds to a second text feature;

[0237] The fusion unit 823 is used to perform weighted fusion of the second image features and the second text features based on the first weight and the second weight to obtain the first target features.

[0238] Optionally, see Figure 9 The first determining unit 813 is used to concatenate the first image features and the first text features to obtain the second target features; to perform spatial transformation on the second target features to determine the first weight; and to determine the second weight based on the first weight, wherein the sum of the first weight and the second weight is 1.

[0239] Optionally, see Figure 9 The transformation module 802 includes:

[0240] The second determining unit 812 is used to determine the product of the first image feature and the first transformation coefficient to obtain the third image feature;

[0241] The third determining unit 822 is used to determine the product of the first text feature and the second transformation coefficient to obtain the third text feature;

[0242] The processing unit 832 is used to process the third image feature and the third text feature based on the activation function to obtain the second image feature and the second text feature.

[0243] Optionally, see Figure 9 The device also includes:

[0244] The third acquisition module 805 is used to acquire the target text;

[0245] The encoding module 806 is used to encode the target text to obtain the first text feature.

[0246] Optionally, see Figure 9 The third acquisition module 805 includes:

[0247] The first acquisition unit 815 is used to acquire the identity information of the target object;

[0248] The second acquisition unit 825 is used to acquire at least one abnormality-related information of the target object, wherein the at least one abnormality-related information is used to reflect the abnormal condition of the target object.

[0249] The generation unit 835 is used to generate target text based on identity information and at least one abnormality-related information.

[0250] Figure 10 This is a flowchart of a health status prediction device provided in an embodiment of this application. See also... Figure 10 The device includes:

[0251] The first acquisition module 1001 is used to acquire a target image and target text, the target image and target text correspond to the same target object, and the target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part.

[0252] Module 1002 is used to call an image processing model based on the target image and target text to obtain abnormal status information of the target object. The image processing model is used to perform spatial transformation on the first image feature of the target image and the first text feature of the target text to obtain a second image feature and a second text feature. The second image feature and the second text feature have the same dimension. The second image feature and the second text feature are fused to obtain a first target feature. Based on the first target feature, abnormal status information of the target object is obtained.

[0253] The image processing apparatus provided in this application embodiment inputs an image of an abnormal part of a target object and target text into an image processing model. The image processing model acquires image features and text features of the same dimension, enabling better fusion of image features and text features. Based on the fused features, abnormal condition information of the target object is obtained, which can improve the accuracy of obtaining abnormal condition information. Furthermore, obtaining abnormal condition information of the target object through the image processing model can improve the efficiency of obtaining abnormal condition information.

[0254] Optionally, see Figure 11 Calling module 1002 includes:

[0255] Determining unit 1012 is used to determine the first text feature of the target text;

[0256] The prediction unit 1022 is used to input the target image and the first text features into the image processing model and output the abnormal situation information of the target object. The image processing model is also used to determine the first image features of the target image and receive the first text features.

[0257] Optionally, see Figure 11 The image processing model includes a network module, a fusion module, and a fully connected layer. The prediction unit 1022 is used to determine the first image features of the target image through the network module; to perform spatial transformation on the first image features and the first text features through the fusion module to obtain the second image features and the second text features, which have the same dimension; to fuse the second image features and the second text features to obtain the first target features; and to obtain the abnormal status information of the target object based on the first target features through the fully connected layer.

[0258] Optionally, see Figure 11 The device also includes:

[0259] The second acquisition module 1003 is used to acquire sample data, which includes sample images and sample target text. The sample images and sample target text correspond to the same sample object, and the sample data labels the actual results of the health status of the sample object.

[0260] The first determining module 1004 is used to determine the fourth text feature of the sample target text;

[0261] The second determining module 1005 is used to determine the fourth image feature of the sample image through the network module;

[0262] The fusion module 1006 is used to perform spatial transformation on the fourth image feature and the fourth text feature to obtain the fifth image feature and the fifth text feature, which have the same dimension; and to fuse the fifth image feature and the fifth text feature to obtain the fourth target feature.

[0263] The third acquisition module 1007 is used to acquire abnormal status information of the target object based on the fourth target feature through the fully connected layer;

[0264] The update module 1008 is used to update the model parameters of the network module and the model parameters of the fusion module based on the acquired results and the actual results, respectively, to obtain the image processing model.

[0265] It should be noted that the health status prediction device provided in the above embodiments is only illustrated by the division of the above functional modules when predicting the health status of the target object. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the target text generation device and the target text generation method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0266] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed in the image processing method of the above embodiments.

[0267] Optionally, the computer device is provided as a terminal. Figure 12 A schematic diagram of the structure of a terminal 1200 provided in an exemplary embodiment of this application is shown.

[0268] Terminal 1200 includes a processor 1201 and a memory 1202.

[0269] Processor 1201 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1201 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1201 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1201 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0270] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1202 are used to store at least one computer program, which is used by the processor 1201 to implement the image processing method provided in the method embodiments of this application.

[0271] In some embodiments, the terminal 1200 may also optionally include a peripheral device interface 1203 and at least one peripheral device. The processor 1201, memory 1202, and peripheral device interface 1203 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1203 via a bus, signal line, or circuit board. Optionally, the peripheral device includes at least one of a radio frequency circuit 1204 and a display screen 1205.

[0272] Peripheral device interface 1203 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1201 and memory 1202. In some embodiments, processor 1201, memory 1202 and peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1201, memory 1202 and peripheral device interface 1203 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0273] The radio frequency (RF) circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1204 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1204 can communicate with other devices via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1204 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0274] Display screen 1205 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1205 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1201 for processing. In this case, display screen 1205 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1205, disposed on the front panel of terminal 1200; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1200 or in a folded design; in still other embodiments, display screen 1205 may be a flexible display screen, disposed on a curved or folded surface of terminal 1200. Furthermore, display screen 1205 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1205 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0275] Those skilled in the art will understand that Figure 12 The structure shown does not constitute a limitation on terminal 1200 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0276] Optionally, the computer device is provided as a server. Figure 13 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1300 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1301 and one or more memories 1302. The memories 1302 store at least one computer program, which is loaded and executed by the processor 1301 to implement the methods provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0277] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed in the image processing method of the above embodiments.

[0278] This application also provides a computer program product, which includes a computer program loaded and executed by a processor to perform the operations performed by the image processing method described above. In some embodiments, the computer program involved in this application can be deployed and executed on a single computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed across multiple locations and interconnected via a communication network. These multiple computer devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0279] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0280] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. An image processing method, characterized in that, The method includes: The first image feature of the target image and the first text feature of the target text are obtained, wherein the target image and the target text correspond to the same target object, and the target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part; The first image feature and the first text feature are spatially transformed to obtain the second image feature and the second text feature, which have the same dimension. The first image feature and the first text feature are concatenated to obtain the second target feature; the second target feature is spatially transformed to determine the first weight; based on the first weight, the second weight is determined, and the sum of the first weight and the second weight is 1; based on the first weight and the second weight, the second image feature and the second text feature are weighted and fused to obtain the first target feature, where the first weight is used to indicate the degree of contribution of the first image feature to the fused first target feature, and the second weight is used to indicate the degree of contribution of the first text feature to the fused first target feature. Based on the first target feature, obtain the abnormal status information of the target object.

2. The method according to claim 1, characterized in that, The step of spatially transforming the first image features and the first text features to obtain second image features and second text features includes: The product of the first image feature and the first transform coefficient is determined to obtain the third image feature; The product of the first text feature and the second transformation coefficient is determined to obtain the third text feature; Based on the activation function, the third image feature and the third text feature are processed respectively to obtain the second image feature and the second text feature.

3. The method according to claim 1, characterized in that, Obtain the first text features of the target text, including: Get the target text; The target text is encoded to obtain the first text feature.

4. The method according to claim 3, characterized in that, The acquisition of the target text includes: Obtain the identity information of the target object; Obtain at least one abnormality-related information of the target object, wherein the at least one abnormality-related information is used to reflect the abnormal condition of the target object; The target text is generated based on the identity information and the at least one abnormal information.

5. An image processing method, characterized in that, The method includes: Acquire a target image and target text, wherein the target image and the target text correspond to the same target object, and the target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part; Based on the target image and the target text, an image processing model is invoked to obtain abnormal status information of the target object. The image processing model is used to perform spatial transformation on the first image feature of the target image and the first text feature of the target text to obtain a second image feature and a second text feature, the second image feature and the second text feature having the same dimension; the first image feature and the first text feature are concatenated to obtain a second target feature; the second target feature is spatially transformed to determine a first weight; based on the first weight, a second weight is determined, the sum of the first weight and the second weight is 1; based on the first weight and the second weight, the second image feature and the second text feature are weighted and fused to obtain a first target feature, the first weight indicating the contribution of the first image feature to the fused first target feature, and the second weight indicating the contribution of the first text feature to the fused first target feature; based on the first target feature, abnormal status information of the target object is obtained.

6. The method according to claim 5, characterized in that, The step of invoking an image processing model based on the target image and the target text to obtain abnormal status information of the target object includes: Determine the first textual feature of the target text; The target image and the first text feature are input into the image processing model, and the abnormal status information of the target object is output. The image processing model is also used to determine the first image feature of the target image and receive the first text feature.

7. The method according to claim 6, characterized in that, The image processing model includes a network module, a fusion module, and a fully connected layer; The network module determines the first image feature of the target image; The fusion module performs spatial transformation on the first image feature and the first text feature to obtain a second image feature and a second text feature, which have the same dimension; the first image feature and the first text feature are then concatenated to obtain a second target feature. The second target feature is spatially transformed to determine a first weight; based on the first weight, a second weight is determined; based on the first weight and the second weight, the second image feature and the second text feature are weighted and fused to obtain the first target feature; The fully connected layer obtains abnormal status information of the target object based on the first target feature.

8. The method according to claim 7, characterized in that, The method further includes: Acquire sample data, which includes sample images and sample target text, wherein the sample images and sample target text correspond to the same sample object, and the sample data annotates the actual health status of the sample object; Determine the fourth text feature of the target text of the sample; The network module determines the fourth image feature of the sample image; The fusion module performs spatial transformation on the fourth image feature and the fourth text feature to obtain a fifth image feature and a fifth text feature, which have the same dimension; the fifth image feature and the fifth text feature are then fused to obtain a fourth target feature. Based on the fourth target feature, the fully connected layer obtains the abnormal status information of the target object; Based on the obtained results and the actual results, the model parameters of the network module and the model parameters of the fusion module are updated respectively to obtain the image processing model.

9. An image processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire the first image features of the target image and the first text features of the target text, wherein the target image and the target text correspond to the same target object, and the target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part; The transformation module is used to perform spatial transformation on the first image feature and the first text feature to obtain a second image feature and a second text feature, wherein the second image feature and the second text feature have the same dimension. The fusion module is used to concatenate the first image feature and the first text feature to obtain a second target feature; perform spatial transformation on the second target feature to determine a first weight; determine a second weight based on the first weight, wherein the sum of the first weight and the second weight is 1; and perform weighted fusion on the second image feature and the second text feature based on the first weight and the second weight to obtain a first target feature, wherein the first weight is used to indicate the degree of contribution of the first image feature to the fused first target feature, and the second weight is used to indicate the degree of contribution of the first text feature to the fused first target feature. The prediction module is used to predict the health status of the target object based on the first target feature.

10. An image processing apparatus, characterized in that, The device includes: The first acquisition module is used to acquire a target image and a target text, wherein the target image and the target text correspond to the same target object, and the target image is an image of an abnormal part of the target object, and the target text records abnormal information related to the abnormal part; The calling module is used to invoke an image processing model based on the target image and the target text to obtain abnormal status information of the target object. The image processing model is used to perform spatial transformation on the first image feature of the target image and the first text feature of the target text to obtain a second image feature and a second text feature, the second image feature and the second text feature having the same dimension; concatenate the first image feature and the first text feature to obtain a second target feature; perform spatial transformation on the second target feature to determine a first weight; determine a second weight based on the first weight, the sum of the first weight and the second weight being 1; perform weighted fusion on the second image feature and the second text feature based on the first weight and the second weight to obtain a first target feature, the first weight being used to indicate the degree of contribution of the first image feature to the fused first target feature, the second weight being used to indicate the degree of contribution of the first text feature to the fused first target feature; and obtain abnormal status information of the target object based on the first target feature.

11. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to perform the operations of the image processing method as described in any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to perform the operations of the image processing method as described in any one of claims 1 to 8.

13. A computer program product, comprising a computer program, characterized in that, The computer program is loaded and executed by a processor to perform the operations of the image processing method as described in any one of claims 1 to 8.

Images