Image processing methods and training methods for image processing models

Abstract

This specification provides an image processing method and an image processing model training method. The image processing method includes: receiving an image processing task, wherein the image processing task carries a target detection region corresponding to a target detection region, and the image processing task is used to detect whether there are anomalies in the target detection region; inputting the target detection region into an image processing model to obtain a target detection result corresponding to the target detection region, wherein the image processing model generates an initial prediction result based on the target detection region, and determines the target detection result based on the initial prediction result and a result relationship matrix, the result relationship matrix being used to identify the correlation between multiple results. The method provided in this specification filters the initial prediction result through the result relationship matrix to obtain a more accurate target detection result.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to an image processing method. Background Technology

[0002] As people's living standards improve, more and more people are paying attention to their health. As the largest organ in the human body, the skin is in direct contact with the external environment. Due to a combination of factors, skin diseases affect nearly one-third of the world's population. However, there is a huge shortage of dermatologists, and many patients with skin diseases cannot get a professional diagnosis.

[0003] With the advent of the big data era, deep learning technology is increasingly being applied to image recognition. By performing image recognition on images of skin patients, it can assist in determining the presence and type of skin lesions in the image. Currently, image classification algorithms for skin diseases simply predict the disease label; that is, given an image containing skin, the output is the predicted disease. However, due to the complexity and diversity of skin diseases, existing algorithms cannot achieve an accuracy rate of over 90% in image recognition. Typically, the top predictions need to be filtered to provide the most accurate result. Existing image recognition methods often result in mutually exclusive predictions, leading to inaccurate predictions. Therefore, how to accurately locate lesions in images and identify their types has become a pressing problem for engineers. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide an image processing method. One or more embodiments of this specification also relate to an image processing apparatus, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an image processing method is provided, comprising:

[0006] The image processing task is received, wherein the image processing task carries an image to be detected corresponding to a target detection region, and the image processing task is used to detect whether there is an anomaly in the target detection region;

[0007] The image to be detected is input into an image processing model to obtain the target detection result corresponding to the target detection region. The image processing model generates an initial prediction result based on the image to be detected and determines the target detection result based on the initial prediction result and the result relationship matrix. The result relationship matrix is ​​used to identify the correlation between multiple results.

[0008] According to a second aspect of the embodiments of this specification, a method for processing skin lesion images is provided, comprising:

[0009] Receive a skin lesion image processing task, wherein the skin lesion image processing task carries a skin lesion image to be detected corresponding to a target detection area, and the skin lesion image processing task is used to detect whether there is an abnormality in the target detection area;

[0010] The image of the skin lesion to be detected is input into the skin lesion image processing model to obtain the target detection result corresponding to the target detection area. The skin lesion image processing model generates an initial prediction result based on the image of the skin lesion to be detected, and determines the target detection result based on the initial prediction result and the result relationship matrix. The result relationship matrix is ​​used to identify the correlation between multiple results.

[0011] According to a third aspect of the embodiments of this specification, a method for training an image processing model is provided, applied to a cloud-based device, comprising:

[0012] Obtain the sample image and the corresponding sample detection result;

[0013] The sample image is input into the image processing model to obtain the predicted detection results, predicted abnormal feature information and predicted non-abnormal feature information output by the image processing model;

[0014] The model loss value is calculated based on the sample detection results, the predicted detection results, the predicted abnormal feature information, and the predicted non-abnormal feature information;

[0015] The model parameters of the image processing model are adjusted according to the model loss value until the model training stopping condition is met, thereby obtaining the model parameters of the image processing model.

[0016] The model parameters of the image processing model are sent to the end-side device.

[0017] According to a fourth aspect of the embodiments of this specification, an image processing method is provided, comprising:

[0018] The system receives an image processing request sent by a user, wherein the image processing request includes an image processing task, the image processing task carries an image to be detected corresponding to a target detection region, and the image processing task is used to detect whether there is an anomaly in the target detection region;

[0019] The image to be detected is input into an image processing model to obtain the target detection result corresponding to the target detection region. The image processing model generates an initial prediction result based on the image to be detected and determines the target detection result based on the initial prediction result and the result relationship matrix. The result relationship matrix is ​​used to identify the correlation between multiple results.

[0020] Send the target detection result corresponding to the target detection area to the user.

[0021] According to a fifth aspect of the embodiments of this specification, an image processing apparatus is provided, comprising:

[0022] The receiving module is configured to receive an image processing task, wherein the image processing task carries an image to be detected corresponding to a target detection region, and the image processing task is used to detect whether there is an anomaly in the target detection region;

[0023] The detection module is configured to input the image to be detected into an image processing model to obtain the target detection result corresponding to the target detection region. The image processing model generates an initial prediction result based on the image to be detected and determines the target detection result based on the initial prediction result and the result relationship matrix. The result relationship matrix is ​​used to identify the correlation between multiple results.

[0024] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0025] Memory and processor;

[0026] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above method.

[0027] According to a seventh aspect of an embodiment of this specification, a computer-readable storage medium is provided that stores computer-executable instructions that, when executed by a processor, implement the steps of the method described above.

[0028] According to an eighth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described method.

[0029] The method provided in one embodiment of this specification, during the processing of the image to be detected, in the image processing model, while processing the image to be detected and obtaining the initial prediction result, also refers to the abnormal feature information generated during the processing. After obtaining the initial prediction result, it continues to refer to a pre-set result relationship matrix to further filter the results in the initial prediction result, avoiding mutually exclusive prediction results, thereby making the final target detection result more accurate. Attached Figure Description

[0030] Figure 1 This is an architecture diagram of an image processing system provided in one embodiment of this specification;

[0031] Figure 2 This is a flowchart of an image processing method provided in one embodiment of this specification;

[0032] Figure 3 This is a schematic diagram of the result relationship matrix provided in one embodiment of this specification;

[0033] Figure 4 This is a flowchart of a skin lesion image processing method provided in one embodiment of this specification;

[0034] Figure 5 This is a schematic diagram of a skin lesion image processing method provided in one embodiment of this specification;

[0035] Figure 6 This is a flowchart of a training method for an image processing model provided in one embodiment of this specification;

[0036] Figure 7 This is a schematic diagram of the structure of an image processing model provided in one embodiment of this specification;

[0037] Figure 8 This is a flowchart of another image processing method provided in one embodiment of this specification;

[0038] Figure 9 This is a flowchart illustrating the processing procedure of an image processing method provided in one embodiment of this specification.

[0039] Figure 10 This is a schematic diagram of the structure of an image processing apparatus provided in one embodiment of this specification;

[0040] Figure 11 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0041] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0042] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0043] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0044] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0045] With the advent of the big data era, deep learning technology is increasingly being applied to image recognition. By performing image recognition on images of skin patients, it can help determine whether skin lesions exist in the image and what type of lesions they are. Currently, image classification algorithms for skin diseases simply predict the disease label; that is, given an image containing skin, the output is the predicted disease. However, due to the complexity and diversity of skin diseases, existing algorithms cannot achieve an accuracy rate of over 90% in image recognition. It is usually necessary to filter the top predictions to provide the most accurate result. Furthermore, existing image recognition methods often result in mutually exclusive predictions, leading to inaccurate predictions.

[0046] Based on this, an image processing method is provided in this specification. This specification also relates to an image processing apparatus, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.

[0047] See Figure 1 , Figure 1 This specification illustrates an architecture diagram of an image processing system according to one embodiment of the present specification. The image processing system may include a client 100 and a server 200.

[0048] Client 100 is used to send an image processing task to server 200. The image processing task carries the image to be detected corresponding to the target detection area. The image processing task is used to detect whether there is an anomaly in the target detection area.

[0049] Server 200 is used to input the image to be detected into an image processing model to obtain the target detection result corresponding to the target detection region. The image processing model generates an initial prediction result based on the image to be detected and determines the target detection result based on the initial prediction result and a result relationship matrix. The result relationship matrix is ​​used to identify the correlation between multiple results. The server 200 sends the target detection result to client 100.

[0050] Client 100 is also used to receive target detection results sent by server 200.

[0051] Using the scheme of the embodiments of this specification, an image processing task is received, wherein the image processing task carries an image to be detected corresponding to a target detection region, and the image processing task is used to detect whether there is an anomaly in the target detection region; the image to be detected is input into an image processing model to obtain a target detection result corresponding to the target detection region, wherein the image processing model generates an initial prediction result based on the image to be detected, and determines the target detection result based on the initial prediction result and a result relationship matrix, wherein the result relationship matrix is ​​used to identify the correlation between multiple results.

[0052] The scheme provided in the embodiments of this specification first generates an initial prediction result based on the image to be detected, and then provides a basis for correcting the initial prediction result based on the result relationship matrix. The result relationship matrix reflects the correlation between multiple results. By using the result relationship matrix, results with large prediction deviations in the initial prediction results are removed, thereby further improving the accuracy of the prediction results.

[0053] An image processing system may include multiple clients 100 and a server 200. Clients 100 can be referred to as edge devices, and the server 200 can be referred to as cloud devices. Multiple clients 100 can establish communication connections through the server 200. In an image processing scenario, the server 200 is used to provide image processing services between the multiple clients 100. Each client 100 can act as a sender or receiver, communicating through the server 200.

[0054] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In image processing scenarios, users can publish data streams to server 200 through client 100, server 200 can generate target detection results based on the data stream, and push the target detection results to other clients that have established communication.

[0055] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0056] Client 100 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as a real-time communication (RTC) SDK. Client 100 can be deployed on electronic devices and depends on the device or certain apps on the device to run. Electronic devices may have displays and support information browsing, such as personal mobile terminals like mobile phones, tablets, and personal computers. Various other types of applications can also be configured on electronic devices, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0057] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0058] It is worth noting that the image processing methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the image processing methods provided in the embodiments of this specification. In other embodiments, the image processing methods provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0059] See Figure 2 , Figure 2 This specification shows a flowchart of an image processing method according to an embodiment, which specifically includes the following steps:

[0060] Step 202: Receive an image processing task, wherein the image processing task carries an image to be detected corresponding to the target detection region, and the image processing task is used to detect whether there is an anomaly in the target detection region.

[0061] In practical applications, image processing tasks sent by users can be received through either a server or a client.

[0062] Specifically, an image processing task is used to detect whether there are anomalies within a target detection region. The image processing task carries the image to be detected corresponding to the target detection region. Furthermore, the target detection region can be understood as the region used to predict whether there are anomalies. For example, the target detection region can be any organ of the human body, such as the mouth, eyes, or local skin. By predicting whether there are anomalies in the target detection region, the prediction results can be used to further assist in judging the state of the object to be detected, thus providing assistance after the state of the object to be detected has been determined.

[0063] It should be noted that, in one or more embodiments of this specification, the image processing task can be applied to the recognition of various medical images and to determine whether there are abnormalities in the target detection area of ​​the medical image based on image features. For example, in the application scenario of eye disease detection, the presence and type of lesions in the eye can be detected based on an eye photograph; in the application scenario of skin disease detection, the presence and type of skin lesions can be detected based on a local skin photograph; thereby helping doctors to make an auxiliary judgment on whether there are abnormalities in the target detection area, thus facilitating subsequent treatment.

[0064] In a specific embodiment provided in this specification, in an application scenario for detecting eye diseases, the image to be detected is an image of the user's eye, specifically, an image taken by the user using a portable device such as a mobile phone or camera. In practical applications, the image sizes acquired by various portable devices vary. In the method provided in this specification, to facilitate subsequent processing, the image carrying the target detection area undergoes image preprocessing to standardize the size of the image to be detected.

[0065] In another specific embodiment provided in this specification, taking the application scenario of detecting skin diseases as an example, an initial image of a localized skin area taken by a user using a mobile phone is acquired. Using the OpenCV toolkit, the initial image is processed to a size of 384*384 pixels to be detected, and the pixel values ​​in the image are normalized. In practical applications, the size of the image to be detected can be set according to the actual application; in this specification, the size of the image to be detected is not limited. By performing image detection processing on the localized skin image to be detected, it is possible to detect whether the user has a skin disease and the type of skin disease.

[0066] The terminal receives an image processing task and can use the image to be detected corresponding to the target detection area carried in the image processing task as input to detect whether there are any abnormalities in the target detection area.

[0067] Step 204: Input the image to be detected into the image processing model to obtain the target detection result corresponding to the target detection region. The image processing model generates an initial prediction result based on the image to be detected and determines the target detection result based on the initial prediction result and the result relationship matrix. The result relationship matrix is ​​used to identify the correlation between multiple results.

[0068] In practical applications, after receiving an image processing task, the image to be detected carried by the task is obtained, and the image to be detected is input into the image processing model for processing. The target detection result corresponding to the target detection region output by the image processing model can be obtained. The target detection result specifically includes information such as whether there is an anomaly in the target detection region, the type of anomaly, and the probability of the anomaly type.

[0069] Specifically, an image processing model can extract feature information of the target detection region from the input image to be detected, and make predictions based on this feature information to obtain an initial prediction result. After obtaining the initial prediction result, it is then compared with the result relationship matrix to determine the final target detection result. In practical applications, the image processing model can be a traditional machine learning model or a deep learning model.

[0070] The result relation matrix specifically refers to representing multiple known results in matrix form, used to identify the relationships between these results. See also Figure 3 , Figure 3 This specification illustrates a schematic diagram of a result relationship matrix provided in one embodiment. Figure 3The result relationship matrix shown contains 49 results, and the correlation between any two results is represented by numbers. Here, 2 indicates autocorrelation, 1 indicates similarity, 0 indicates unknown relationship, and -1 indicates mutual exclusion. For example, for result 0 and result 2, the result relationship matrix shows that the relationship between these two results is -1, which means that result 0 and result 2 are mutually exclusive. If result 0 is predicted, the probability of predicting result 2 is relatively small.

[0071] In the embodiments provided in this specification, the result relation matrix plays a role in correcting and optimizing the image processing model to generate the final target detection result. Through this result relation matrix, the target detection result can be made more accurate.

[0072] Specifically, in one embodiment provided in this specification, the image processing model includes an embedding layer, a feature processing layer, a classification layer, and a comparison layer;

[0073] The image to be detected is input into the image processing model to obtain the target detection result corresponding to the target detection region, including S2042-S2048:

[0074] S2042. Input the image to be detected into the embedding layer to obtain the embedded image features.

[0075] The embedding layer is used to embed the input image to be detected, obtaining embedded image features. Image embedding is the process of converting data into a fixed-size feature representation for easier processing and computation. Embedding is an encoding technique that uses low-dimensional vector representations, which, through neural network optimization, can express the relationships between images. Through embedding, images can be encoded into feature vectors that computers can recognize and process.

[0076] Furthermore, the image to be detected is input into the embedding layer to obtain embedded image features, including:

[0077] Based on preset segmentation information, the image to be detected is segmented into multiple sub-images to be detected;

[0078] The preset classifier and multiple sub-images to be detected are concatenated into the input image;

[0079] The image to be input is input into the embedding layer to obtain embedded image features.

[0080] In practical applications, in order to more accurately locate abnormal locations, the image to be detected can be divided into multiple sub-images to be detected, and each sub-image can be processed for recognition. This allows for more accurate location of abnormal locations in the image to be detected during subsequent recognition processes.

[0081] In the embodiments provided in this specification, the preset segmentation information specifically refers to the pre-defined information for segmenting the image to be detected, such as segmenting the image to be detected into 16*16 sub-images or into 32*32 sub-images. The preset segmentation information is a pre-set hyperparameter that can be set according to actual conditions.

[0082] Based on the preset segmentation information, the image to be detected can be segmented into a corresponding number of sub-images to be detected. For example, if the preset segmentation information is 16*16, then the image to be detected (384*384) will be segmented into 576 sub-images to be detected based on this segmentation information. Each sub-image to be detected is marked according to its position information in the image to be detected, so that multiple sub-images to be detected can be restored later based on this position information.

[0083] The preset classifier specifically refers to the "CLS" identifier. "CLS" is used in classification tasks. In practical applications, the information input into the image processing model is prefixed with the [CLS] identifier for classification during the model's processing. Taking 576 sub-images to be detected as an example, after adding the preset classifier CLS, the images are input into the embedding layer for embedding processing, obtaining the embedded image features output by the embedding layer. Assuming the embedding layer has a dimension of 768, the embedded image features will have a dimension of 577*768.

[0084] S2044. Input the embedded image features into the feature processing layer to obtain image decoding features and abnormal feature information.

[0085] The feature processing layer is specifically used to process the embedded image features. In practical applications, the image processing model can be a model built based on the Vision Transformer model. The feature processing layer can be understood as multiple Transformer Layers. The embedded image features are sequentially input into multiple consecutive Transformer Layers to obtain image decoding features and anomaly feature information.

[0086] Image decoding features specifically refer to the image feature information obtained after encoding and decoding the image to be detected. Abnormal feature information indicates abnormal feature information in the image to be detected.

[0087] In practical applications, the embedded image features are sequentially input into multiple consecutive Transformer Layers for processing to obtain each self-attention matrix A in each Layer. Initial correlation features are extracted from each self-attention matrix A. The initial correlation features represent the correlation between the CLS in the self-attention matrix and each sub-image to be detected. In practical applications, the first row vector of each self-attention matrix A is taken out and the first element of the first row vector is removed to obtain the initial correlation features of each attention matrix A.

[0088] After weighted averaging of multiple initial correlation features, the target correlation feature can be obtained. This target correlation feature is used to identify the location information of the abnormal position after multiple encoding and decoding processes. The target correlation feature corresponds one-to-one with multiple sub-images to be detected. By restoring the target correlation feature to the image to be detected based on the location information of each sub-image to be detected, the location information of the abnormality can be determined.

[0089] After determining the anomaly location information, the anomaly feature information can be determined based on the anomaly location information and image decoding features. For details, please refer to Formula 1 below:

[0090]

[0091] Among them, f lesion f represents abnormal feature information. non-lesion Indicates non-abnormal feature information, Information indicating the salience of abnormal locations. This represents the image decoding features.

[0092] Using Formula 1 above, the abnormal feature information can be calculated. This abnormal feature information is used to provide corresponding feature information for the initial prediction results in the subsequent classification process.

[0093] S2046. Input the image decoding features and the abnormal feature information into the classification layer to obtain the initial prediction result.

[0094] After obtaining the abnormal feature information, the image decoding features and the abnormal feature information can be input into the classification layer for feature classification, thereby obtaining the initial prediction result.

[0095] In practical applications, the pre-added CLS features are used for classification. Specifically, the classification feature information corresponding to the preset classifier CLS is extracted from the image decoding features, and the classification feature information and the anomaly feature information are concatenated to obtain the concatenated feature information. The concatenated feature information is then processed in the classification layer to obtain the initial prediction result.

[0096] In the method provided in this specification, image decoding features and anomaly feature information are input into the classification layer. In the classification layer, prediction is performed based on the classification feature information and anomaly feature information in the image decoding features to obtain an initial prediction result. Specifically, the initial prediction result includes multiple prediction sub-results and the probability of each prediction sub-result. The prediction sub-results are sorted according to their probabilities, and a first preset number of sub-results are selected as the initial prediction result.

[0097] For example, in a method for detecting skin diseases, based on image decoding features and abnormal feature information, 49 predictive sub-results and their corresponding probabilities are obtained. The probabilities of each predictive sub-result are then sorted, and the top 10 predictive sub-results are selected as the initial prediction results.

[0098] S2048. Input the initial prediction result into the comparison layer to obtain the target detection result, wherein the comparison layer includes a result relationship matrix.

[0099] The alignment layer is specifically used to correct the initial prediction results. It stores the result relationship matrix. Furthermore, the alignment layer corrects the initial prediction results based on the result relationship matrix to obtain the final target detection result.

[0100] In one specific embodiment provided in this specification, the initial prediction result includes multiple initial prediction sub-results;

[0101] The initial prediction result is input into the comparison layer to obtain the target detection result, including:

[0102] Determine the initial prediction sub-result to be processed and at least one reference initial prediction sub-result, wherein the initial prediction sub-result to be processed is any one of a plurality of initial prediction sub-results;

[0103] Based on the result relationship matrix, the initial prediction sub-result to be processed, and each reference initial prediction sub-result, calculate the correlation score corresponding to the initial prediction sub-result to be processed;

[0104] The initial prediction results are sorted based on the correlation scores corresponding to each initial prediction sub-result to be processed, and the target detection results are determined.

[0105] In the above steps, it can be determined that the initial prediction result includes multiple initial prediction sub-results. In the comparison layer, each initial prediction sub-result is voted on according to the result relationship matrix to obtain the correlation score of each initial prediction sub-result. Finally, the final target detection result is determined based on each correlation score.

[0106] In practical applications, the initial prediction sub-result to be processed is first determined from multiple initial prediction sub-results, and other initial prediction sub-results besides the one to be processed are determined as reference initial prediction sub-results. Specifically, the initial prediction sub-result to be processed refers to the initial prediction sub-result for which the correlation score needs to be determined in the current calculation.

[0107] For example, the initial prediction results include five initial prediction sub-results: initial prediction sub-result 1, initial prediction sub-result 2, initial prediction sub-result 3, initial prediction sub-result 4, and initial prediction sub-result 5. When calculating the correlation score of initial prediction sub-result 1, initial prediction sub-result 1 is the initial prediction sub-result to be processed, and initial prediction sub-results 2-5 are reference initial prediction sub-results. The correlation score corresponding to the initial prediction sub-result to be processed is then calculated by combining the result relationship matrix, the initial prediction sub-result to be processed, and each reference initial prediction sub-result.

[0108] Specifically, based on the result relationship matrix, the initial prediction sub-result to be processed, and each reference initial prediction sub-result, the correlation score corresponding to the initial prediction sub-result to be processed is calculated, including:

[0109] Based on the result relationship matrix, calculate the correlation sub-score between the initial prediction sub-result to be processed and each reference initial prediction sub-result;

[0110] The correlation score corresponding to the initial prediction sub-result to be processed is determined based on the scores of each correlation sub-result.

[0111] In practical applications, the result relationship matrix includes the correlation between any two results. If two results are similar, it is represented by 1; if the relationship between two results is unknown, it is represented by 0; if two results are mutually exclusive, it is represented by -1. Continuing with the previous example, taking initial prediction sub-result 1 as the initial prediction sub-result to be processed, and initial prediction sub-results 2-5 as reference initial prediction sub-results, by querying the result relationship matrix, the correlation sub-scores between initial prediction sub-result 1 and each reference initial prediction sub-result are determined in turn. Then, the correlation sub-scores are added together to obtain the correlation score corresponding to initial prediction sub-result 1.

[0112] Based on the same method described above, the correlation score of each initial prediction sub-result is calculated, and then the initial prediction sub-results are re-sorted according to their correlation scores. A preset number of results are selected as the target detection results.

[0113] For example, if there are 10 initial prediction sub-results, after reordering these 10 sub-results according to their correlation scores, the top 5 sub-results are selected as the target detection results. In practical applications, the number of sub-results in the initial detection results is greater than the number of sub-results in the target detection results.

[0114] After determining the target detection result, it is also necessary to determine the probability of each sub-result within the target detection result. Specifically, this is calculated based on the relevance score of each sub-result and the total relevance score of all sub-results in the target detection result. For example, if there are 5 results in the target detection result: result 1 has a relevance score of 10; result 2 has a relevance score of 8; result 3 has a relevance score of 7; result 4 has a relevance score of 6; and result 5 has a relevance score of 5, the total score of the 5 results is 36. The probability of result 1 is 10 / 36 * 100% = 27%, and so on. This allows us to determine each sub-result in the target detection result and the probability of each sub-result.

[0115] In one specific embodiment provided in this specification, during the processing of the image to be detected, the image processing model, while processing the image to be detected and obtaining the initial prediction result, also refers to the abnormal feature information generated during the processing. After obtaining the initial prediction result, it continues to refer to a pre-set result relationship matrix to further filter the results in the initial prediction result, avoiding mutually exclusive prediction results, thereby making the final target detection result more accurate.

[0116] In one specific embodiment provided in this specification, the image processing model is obtained through training via the following steps S2062-S2068:

[0117] S2062. Obtain the sample image and the sample detection result corresponding to the sample image.

[0118] Specifically, the sample image refers to the image used to train the image processing model, and the sample detection result refers to the known detection result corresponding to the sample image. In practical applications, taking a skin disease image as an example, the sample detection result is the doctor's diagnosis. The sample image and the sample detection result constitute a sample pair, which is used for weakly supervised training of the image processing model.

[0119] Similar to the processing method for the images to be tested, sample images typically need to be acquired using portable devices and then optimized. Taking the diagnosis of a skin disease patient as an example, images of the patient's skin lesions are taken using devices such as mobile phones or cameras, and the doctor's diagnosis is obtained as the sample test result. It is important to note that obtaining the patient's skin lesion images requires the patient's authorization and consent, and obtaining the doctor's diagnosis also requires the authorization and consent of both the patient and the doctor.

[0120] After obtaining the skin lesion images, blurry and invalid images must be filtered out. After retaining the valid skin lesion images, the images are scaled to a preset size using the OpenCV toolkit, at which point the sample images are obtained.

[0121] In the method provided in the embodiments of this specification, the set of sample images used for training is divided into a training set and a test set. The image processing model is trained using the training set and the processing effect of the image processing model is verified using the test set.

[0122] S2064. Input the sample image into the image processing model to obtain the prediction detection result, predicted abnormal feature information and predicted non-abnormal feature information output by the image processing model.

[0123] In the method provided in the embodiments of this specification, after obtaining the training samples, the sample images are input into the image processing model according to the preset training batch. At this time, the image processing model is an untrained image processing model. In the image processing model, the sample images are first processed to obtain the prediction detection results corresponding to the sample images. At the same time, the predicted abnormal feature information and the predicted non-abnormal feature information are also obtained.

[0124] Specifically, the predicted detection result refers to the image processing prediction result obtained after processing by the image processing model obtained through the method provided in this specification. Predicted anomalous feature information specifically refers to feature information extracted from the sample image that may belong to anomalous targets, while predicted non-anomalous feature information specifically refers to feature information extracted from the sample image that may not belong to anomalous targets.

[0125] In practical applications, the sample image is first segmented into multiple sample sub-images. An image processing model is then used to determine which sample sub-image contains anomalies, and the image features of the anomaly-containing sub-images are extracted as predicted anomaly features. Similarly, it is also determined which sample sub-images do not contain anomalies, and the image features of the anomaly-free sub-images are extracted as predicted non-anomaly features.

[0126] Specifically, in one embodiment provided in this specification, the image processing model includes an embedding layer, a feature processing layer, a classification layer, and a comparison layer;

[0127] The sample image is input into an image processing model to obtain the predicted detection results, predicted abnormal feature information, and predicted non-abnormal feature information output by the image processing model, including:

[0128] The sample image is input into the embedding layer to obtain the embedded sample image features;

[0129] The embedded sample image features are input into the feature processing layer to obtain sample image decoding features, predicted abnormal feature information, and predicted non-abnormal feature information;

[0130] The sample image decoding features and the predicted anomaly feature information are input into the classification layer to obtain the prediction detection result.

[0131] In practical applications, the image processing model operates similarly to its application phase during training. The image processing model includes an embedding layer, a feature processing layer, a classification layer, and an alignment layer. The sample image is first input into the embedding layer to obtain embedded sample image features. The processing method for the sample image in the embedding layer is described above in the section on processing the image to be detected, and will not be repeated here.

[0132] In the process of processing the embedded sample image features, the feature processing layer in the embodiments of this specification obtains not only the sample image decoding features, but also the predicted abnormal feature information and the predicted non-abnormal feature information. The predicted abnormal feature information and the predicted non-abnormal feature information are used to compare and learn the abnormal features and non-abnormal features in the target detection area, so that the image processing model can learn to distinguish between abnormal targets and normal areas, and thus can more accurately locate the abnormal location points in the sample image.

[0133] In practical applications, sample images are usually captured by portable devices, and their backgrounds are often complex, containing many irrelevant environmental factors. This makes it difficult for image processing models to accurately locate anomalies. Currently, sample image processing typically uses supervised training, which involves labeling anomalies in the sample images so that the image processing model can identify them. However, supervised training requires a large amount of labeled data, which consumes significant human and material resources.

[0134] The image processing method provided in this manual uses weakly supervised training, employing sample images and their corresponding detection results as sample pairs. It does not precisely mark the location information of anomalies in the sample images. Instead, it processes the sample images through an image processing model to obtain predicted anomaly and non-anomaly features. By comparing and learning between these predicted anomaly and non-anomaly features, precise anomaly localization can be achieved.

[0135] In one specific embodiment provided in this specification, the feature processing layer includes at least one attention sublayer;

[0136] The embedded sample image features are input into the feature processing layer to obtain sample image decoding features, predicted anomaly feature information, and predicted non-anomaly feature information, including:

[0137] The embedded image features are sequentially input into each attention sub-layer to obtain the sample image decoding features and the attention feature matrix corresponding to each attention sub-layer;

[0138] Initial association features corresponding to each attention feature matrix are extracted from each attention feature matrix, and target association features are determined based on each initial association feature. The initial association features represent the weight of abnormal targets in each sample sub-image, and the target association features include the distribution weight of abnormal targets in the sample images.

[0139] Based on the target correlation characteristics, predict abnormal feature information and predict non-abnormal feature information are determined.

[0140] Specifically, taking the scenario of skin lesion detection as an example, the sample image obtained in the method provided in this manual is a clinical image of a skin patient and the doctor's diagnosis result. The clinical image is preprocessed to obtain the sample image, and the diagnosis result is used as the sample detection result for the sample image.

[0141] The sample image after image preprocessing has a size of 384*384. It is then segmented according to a preset segmentation size, resulting in 576 non-overlapping sub-images, each with a size of 16*16. Each sub-image is then processed through an embedding layer to obtain its corresponding embedded sub-image features, each with a dimension of 1*768. CLS tokens are also introduced for classification; these tokens, after embedding, also have a dimension of 1*768.

[0142] Based on the position information of each sample sub-image on the original sample image, Relative PositionEmbedding is used to obtain the positional encoding of each sub-image. The positional encoding of the CLS token is initialized as a 768-dimensional vector of all zeros. The positional encoding and each embedded sub-image are concatenated to obtain the embedded image features.

[0143] The embedded image features are input into a feature processing layer, which consists of multiple consecutive Transformer Layers. After processing by each Transformer Layer, the CLS token and the output features of each embedded sub-image are obtained. The output features of the CLS token are represented by f. cls This indicates that the output features of each embedded sub-image are represented by... This indicates that the value of i ranges from 0 to 576. f cls and f patch These constitute the decoding features of the sample image.

[0144] Simultaneously, the attention feature matrices from the attention sublayers of the Transformer Layer are extracted to determine the final predicted anomaly and non-anomaly features. Specifically, the first row vector of each attention feature matrix is ​​extracted, and the first element is removed to obtain the initial correlation features. The feature dimension of the initial correlation features is 1*576. Since the first row vector in each attention feature matrix represents the relationship information between the CLS token and each other sample sub-image, and the first element is the information of the CLS token itself, extracting the first row vector and then removing the first element can obtain the initial correlation features between each sample sub-image.

[0145] Taking a Transformer Layer with 12 layers as an example, 12 initial correlation features can be obtained. By taking a weighted average, the target correlation features can be obtained.

[0146] The target correlation features are reconstructed using the location information of each sample sub-image to obtain the localization feature information of the abnormal target. (i.e., the salience of abnormal targets). The output features of each embedded sub-image are reconstructed using the positional information of each sample sub-image to obtain the image decoding features.

[0147] The predicted anomaly feature information f is calculated using Formula 1 above. lesion and predicting non-abnormal feature information f non-lesion .

[0148] S2066. Calculate the model loss value based on the sample detection results, the predicted detection results, the predicted abnormal feature information, and the predicted non-abnormal feature information.

[0149] After obtaining the predicted detection results, the predicted abnormal feature information, and the predicted non-abnormal feature information, and combining them with the sample detection results, the model loss value can be calculated. There are many methods for calculating the model loss value, such as the cross-entropy loss function, the maximum loss function, the average loss function, etc. In this specification, the specific method of the loss function is not limited, and the actual application shall prevail.

[0150] In another specific embodiment provided in this specification, the model loss value is calculated based on the sample detection result, the predicted detection result, the predicted abnormal feature information, and the predicted non-abnormal feature information, including:

[0151] Calculate the first loss value based on the sample detection results and the predicted detection results;

[0152] Calculate the second loss value based on the predicted abnormal feature information and the predicted non-abnormal feature information;

[0153] The model loss value is determined based on the first loss value and the second loss value.

[0154] In practical applications, image processing models need to accurately locate anomalies and determine prediction results based on the information from these anomalies. Therefore, anomaly locations are determined by comparing predicted and predicted non-anomaly features. Based on this, a first loss value is calculated using the sample detection results and the predicted detection results, as detailed in Formula 2 below.

[0155] L d =-ylog(p d )Formula 2

[0156] Among them, L d Let y be the first loss value, y be the sample detection result, and p be the second loss value. d This refers to the predicted detection result. Using Formula 2 above, the first loss value can be calculated based on the sample detection result and the predicted detection result. It's important to note that the predicted detection result p obtained during training... d This refers to detection results that have not been reordered using the result relation matrix. It is similar to the initial prediction results of the model in actual applications, rather than the target detection results of the model in actual applications.

[0157] The second loss value is calculated based on the predicted abnormal feature information and the predicted non-abnormal feature information, as shown in Formula 3 below:

[0158] L l = -log(1-sim(f) lesion ,f non-lesion )) Formula 3

[0159] Specifically, L l For the second loss value, f lesion To predict anomalous feature information, f non-lesion This is used to predict non-abnormal feature information. `sim` is the cosine similarity. By comparing and learning the loss function between predicted and predicted anomalous feature information, the model can more accurately locate anomalous points.

[0160] After obtaining the first and second loss values, the model loss value can be determined using the following formula 4.

[0161] L = L l +L d Formula 4

[0162] Where L is the model loss value.

[0163] S2068. Adjust the model parameters of the image processing model according to the model loss value until the model training stops.

[0164] After obtaining the model loss value, the model parameters of the image processing model can be adjusted based on the model loss value. Specifically, the model parameters of the image processing model can be updated sequentially by backpropagating the model loss value.

[0165] After adjusting the model parameters, the above steps can be repeated to continue training the image processing model until the training stopping condition is met. In practical applications, the training stopping condition for the image processing model includes:

[0166] The model loss value is less than a preset threshold; and / or

[0167] The training rounds have reached the preset number of training rounds.

[0168] Specifically, during the training of the image processing model, the training stopping condition can be set to the model loss value being less than a preset threshold, or the training stopping condition can be set to the number of training rounds as preset, such as 10 rounds. In this manual, the preset threshold for the loss value and / or the preset number of training rounds are not specifically limited, and the actual application shall prevail.

[0169] The method provided in the embodiments of this specification uses comparative learning during the training of the image processing model, which improves the accuracy of anomaly localization, especially for small anomalies, making the location of anomalies more accurate. Simultaneously, it fuses global image features and local anomaly features, making full use of image information to achieve more accurate classification of the prediction results.

[0170] See Figure 4 , Figure 4 This specification illustrates a flowchart of a skin lesion image processing method according to an embodiment, which specifically includes the following steps:

[0171] Step 402: Receive a skin lesion image processing task, wherein the skin lesion image processing task carries a skin lesion image to be detected corresponding to the target detection area, and the skin lesion image processing task is used to detect whether there is an abnormality in the target detection area.

[0172] Step 404: Input the image of the skin lesion to be detected into the skin lesion image processing model to obtain the target detection result corresponding to the target detection area. The skin lesion image processing model generates an initial prediction result based on the image of the skin lesion to be detected, and determines the target detection result based on the initial prediction result and the result relationship matrix. The result relationship matrix is ​​used to identify the correlation between multiple results.

[0173] It should be noted that the implementation methods of steps 402 and 404 are the same as those of steps 202-204 above, and will not be described again in the embodiments of this specification.

[0174] For example, in a skin disease detection application scenario, the target detection area is a localized area of ​​a patient's skin. The goal is to detect whether a skin disease exists in this area and, if so, the type of skin disease. A skin lesion image processing task is received, which includes an image of the skin lesion to be detected corresponding to the target user's localized skin. This image is acquired using a portable device such as a mobile phone or camera. This skin lesion image processing task is used to detect whether the user has a skin disease.

[0175] The skin lesion image processing model used in the embodiments of this specification is the image processing model in the above embodiments. The model structure of the skin lesion image processing model is the same as that of the image processing model in the above embodiments, and will not be described again here. By inputting the skin lesion image to be detected into the skin lesion image processing model, the detection result corresponding to the local skin output by the image processing model can be obtained, thereby realizing the automatic detection of whether there is a skin disease in the local skin and the type of skin disease.

[0176] See Figure 5 , Figure 5 A schematic diagram of a skin lesion image processing method provided in one embodiment of this specification is shown. For example... Figure 5 As shown, the image of the skin lesion to be detected is input into the skin lesion image processing model. In the skin lesion image processing model, after processing by the embedding layer and the feature processing layer, the image decoding features are obtained. After processing by the classification layer, the initial prediction result is obtained. The initial prediction result is input into the comparison layer, where it is compared with the result relationship matrix to obtain the final target prediction result.

[0177] The skin lesion image processing method provided in the embodiments of this specification, in the skin lesion image processing model, processes the skin lesion image to be detected and, in the process of obtaining the initial prediction result, also refers to the abnormal feature information generated during the processing. After obtaining the initial prediction result, it continues to refer to a pre-set result relationship matrix to further filter the results in the initial prediction result, avoiding mutually exclusive prediction results, thereby making the final target detection result more accurate.

[0178] See Figure 6 , Figure 6 The flowchart illustrates a training method for an image processing model according to an embodiment of this specification, applied to a cloud-side device, and specifically includes the following steps:

[0179] Step 602: Obtain the sample image and the sample detection result corresponding to the sample image.

[0180] Step 604: Input the sample image into the image processing model to obtain the prediction detection results, predicted abnormal feature information and predicted non-abnormal feature information output by the image processing model.

[0181] Step 606: Calculate the model loss value based on the sample detection results, the predicted detection results, the predicted abnormal feature information, and the predicted non-abnormal feature information.

[0182] Step 608: Adjust the model parameters of the image processing model according to the model loss value until the model training stopping condition is met, and obtain the model parameters of the image processing model.

[0183] Step 610: Send the model parameters of the image processing model to the end device.

[0184] It should be noted that steps 602 and 608 are implemented in the same way as S2062-S2068 above, and will not be described again in the embodiments of this specification.

[0185] During the training of an image processing model via a cloud-based device, if the trained image processing model needs to be deployed on the cloud-based device, a notification indicating that the model training is complete can be sent to the user; if the trained image processing model is deployed on the edge device, the model parameters of the image processing model can be sent to the user.

[0186] See Figure 7 , Figure 7 A schematic diagram of the structure of an image processing model provided in one embodiment of this specification is shown, as follows: Figure 7 As shown, the sample image is segmented into multiple sample sub-images and input into Patch Embedding for processing. A [CLS] token is also introduced for subsequent classification. After performing Position Embedding on each sample sub-image, the embedded sample image features are obtained. These embedded sample image features are then input into multiple TransformerLayers for processing to obtain the output features f of the CLS token. cls Simultaneously, the embedded sub-image features of each sample sub-image are restored to obtain the image decoding features. Localization feature information is obtained based on the attention matrix of the Transformer Layer. And through image decoding features and location feature information Calculate the predicted anomaly feature information f lesion and predicting non-abnormal feature information f non-lesion .

[0187] Through f_cls and f lesion After fusion, the data is input into a classification system for classification, yielding predicted detection results. The first loss value L is then calculated based on the predicted detection results and the sample detection results. d Based on the predicted anomaly feature information f lesion and predicting non-abnormal feature information f non-lesion Calculate the second loss value L f .

[0188] The model loss value is calculated using the first loss value and the second loss value, and the model parameters of the image processing model are adjusted by backpropagation based on the model loss value, thereby achieving model training of the image processing model.

[0189] In practical applications, training a model requires a large amount of data and significant computing resources, which edge devices may lack. Therefore, the model training process can be implemented on cloud devices. After obtaining the model parameters of the image processing model, the cloud device can send these parameters to the edge device. The edge device can then build the image processing model locally based on these parameters and further utilize the model for image processing.

[0190] The method provided in the embodiments of this specification uses comparative learning during the training of the image processing model, which improves the accuracy of anomaly localization, especially for small anomalies, making the location of anomalies more accurate. Simultaneously, it fuses global image features and local anomaly features, making full use of image information to achieve more accurate classification of the prediction results.

[0191] See Figure 8 , Figure 8 This specification shows a flowchart of an image processing method according to an embodiment, which specifically includes the following steps:

[0192] Step 802: Receive an image processing request sent by the user, wherein the image processing request includes an image processing task, the image processing task carries the image to be detected corresponding to the target detection area, and the image processing task is used to detect whether there is an anomaly in the target detection area.

[0193] Step 804: Input the image to be detected into the image processing model to obtain the target detection result corresponding to the target detection region. The image processing model generates an initial prediction result based on the image to be detected and determines the target detection result based on the initial prediction result and the result relationship matrix. The result relationship matrix is ​​used to identify the correlation between multiple results.

[0194] Step 806: Send the target detection result corresponding to the target detection area to the user.

[0195] It should be noted that the specific implementation methods of steps 802-804 are the same as those of steps 202-204 above, and will not be repeated in the embodiments of this specification.

[0196] In this embodiment, an image processing request sent by a user is received. The image processing request includes an image processing task. After the image processing is completed by the image processing method described above and the target detection result is obtained, the target detection result needs to be returned to the user so that the user can perform corresponding follow-up processing based on the target detection result.

[0197] The method provided in the embodiments of this specification, during the processing of the image to be detected, in the image processing model, while processing the image to be detected and obtaining the initial prediction result, also refers to the abnormal feature information generated during the processing. After obtaining the initial prediction result, it continues to refer to a pre-set result relationship matrix to further filter the results in the initial prediction result, avoiding mutually exclusive prediction results, thereby making the final target detection result more accurate.

[0198] The following is in conjunction with the appendix Figure 9 Taking the image processing method provided in this specification in the detection of skin diseases as an example, the image processing method will be further explained. Among other things, Figure 9 This specification illustrates a flowchart of an image processing method according to an embodiment, which specifically includes the following steps:

[0199] Step 902: Acquire an image of the patient's skin lesion area and scale the image of the skin lesion area to a preset size to obtain an image to be detected, wherein the image of the skin lesion area is obtained by taking a picture with a mobile phone.

[0200] Step 904: Input the image to be detected into the pre-trained image processing model. In the image processing model, obtain the global image features, local image features, and skin lesion distribution weight information of the image to be detected.

[0201] Step 906: Based on local image features and skin lesion distribution weight information, obtain predicted skin lesion feature information and predicted non-skin lesion feature information.

[0202] Step 908: Fuse the global image features F and the predicted skin lesion features, and input the fused feature information into the classifier Classfier of the image processing model to obtain the initial prediction result of the image to be detected.

[0203] Step 910: Input the initial comparison results into the comparison layer of the image processing model. The comparison layer includes a preset result relationship matrix, which is a pre-constructed similarity relationship matrix between known skin disease diagnosis results.

[0204] Step 912: In the comparison layer, the initial comparison results are voted on according to the result relationship matrix to exclude mutually exclusive prediction results and obtain the target detection results.

[0205] Corresponding to the above method embodiments, this specification also provides embodiments of an image processing apparatus. Figure 10 A schematic diagram of the structure of an image processing apparatus provided in one embodiment of this specification is shown. Figure 10 As shown, the device includes:

[0206] The receiving module 1002 is configured to receive an image processing task, wherein the image processing task carries an image to be detected corresponding to a target detection region, and the image processing task is used to detect whether there is an anomaly in the target detection region;

[0207] The detection module 1004 is configured to input the image to be detected into an image processing model to obtain the target detection result corresponding to the target detection region. The image processing model generates an initial prediction result based on the image to be detected and determines the target detection result based on the initial prediction result and the result relationship matrix. The result relationship matrix is ​​used to identify the correlation between multiple results.

[0208] Optionally, the image processing model includes an embedding layer, a feature processing layer, a classification layer, and a comparison layer;

[0209] The detection module 1004 is further configured as follows:

[0210] The image to be detected is input into the embedding layer to obtain the embedded image features;

[0211] The embedded image features are input into the feature processing layer to obtain image decoding features and anomaly feature information;

[0212] The image decoding features and the anomaly feature information are input into the classification layer to obtain an initial prediction result;

[0213] The initial prediction result is input into the comparison layer to obtain the target detection result, wherein the comparison layer includes a result relationship matrix.

[0214] Optionally, the detection module 1004 is further configured to:

[0215] Based on preset segmentation information, the image to be detected is segmented into multiple sub-images to be detected;

[0216] The preset classifier and multiple sub-images to be detected are concatenated into the input image;

[0217] The image to be input is input into the embedding layer to obtain embedded image features.

[0218] Optionally, the initial prediction result includes multiple initial prediction sub-results;

[0219] The detection module 1004 is further configured as follows:

[0220] Determine the initial prediction sub-result to be processed and at least one reference initial prediction sub-result, wherein the initial prediction sub-result to be processed is any one of a plurality of initial prediction sub-results;

[0221] Based on the result relationship matrix, the initial prediction sub-result to be processed, and each reference initial prediction sub-result, calculate the correlation score corresponding to the initial prediction sub-result to be processed;

[0222] The initial prediction results are sorted based on the correlation scores corresponding to each initial prediction sub-result to be processed, and the target detection results are determined.

[0223] Optionally, the detection module 1004 is further configured to:

[0224] Based on the result relationship matrix, calculate the correlation sub-score between the initial prediction sub-result to be processed and each reference initial prediction sub-result;

[0225] The correlation score corresponding to the initial prediction sub-result to be processed is determined based on the scores of each correlation sub-result.

[0226] Optionally, the device further includes a training module configured to:

[0227] Obtain the sample image and the corresponding sample detection result;

[0228] The sample image is input into the image processing model to obtain the predicted detection results, predicted abnormal feature information and predicted non-abnormal feature information output by the image processing model;

[0229] The model loss value is calculated based on the sample detection results, the predicted detection results, the predicted abnormal feature information, and the predicted non-abnormal feature information;

[0230] The model parameters of the image processing model are adjusted based on the model loss value until the model training stops.

[0231] Optionally, the image processing model includes an embedding layer, a feature processing layer, a classification layer, and a comparison layer;

[0232] The training module is further configured as follows:

[0233] The sample image is input into the embedding layer to obtain the embedded sample image features;

[0234] The embedded sample image features are input into the feature processing layer to obtain sample image decoding features, predicted abnormal feature information, and predicted non-abnormal feature information;

[0235] The sample image decoding features and the predicted anomaly feature information are input into the classification layer to obtain the prediction detection result.

[0236] Optionally, the feature processing layer includes at least one attention sublayer;

[0237] The training module is further configured as follows:

[0238] The embedded image features are sequentially input into each attention sub-layer to obtain the sample image decoding features and the attention feature matrix corresponding to each attention sub-layer;

[0239] Initial association features corresponding to each attention feature matrix are extracted from each attention feature matrix, and target association features are determined based on each initial association feature. The initial association features represent the weight of abnormal targets in each sample sub-image, and the target association features include the distribution weight of abnormal targets in the sample images.

[0240] Based on the target correlation characteristics, predict abnormal feature information and predict non-abnormal feature information are determined.

[0241] Optionally, the training module is further configured to:

[0242] Calculate the first loss value based on the sample detection results and the predicted detection results;

[0243] Calculate the second loss value based on the predicted abnormal feature information and the predicted non-abnormal feature information;

[0244] The model loss value is determined based on the first loss value and the second loss value.

[0245] The apparatus provided in one embodiment of this specification, during the processing of an image to be detected, in the image processing model, while processing the image to be detected and obtaining an initial prediction result, also refers to abnormal feature information generated during the processing. After obtaining the initial prediction result, it continues to refer to a pre-set result relationship matrix to further filter the results in the initial prediction result, avoiding mutually exclusive prediction results, thereby making the final target detection result more accurate.

[0246] The above is an illustrative scheme of an image processing apparatus according to this embodiment. It should be noted that the technical solution of this image processing apparatus and the technical solution of the image processing method described above belong to the same concept. For details not described in detail in the technical solution of the image processing apparatus, please refer to the description of the technical solution of the image processing method described above.

[0247] Figure 11 A structural block diagram of a computing device 1100 according to one embodiment of this specification is shown. The components of the computing device 1100 include, but are not limited to, a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 via a bus 1130, and a database 1150 is used to store data.

[0248] The computing device 1100 also includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1140 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0249] In one embodiment of this specification, the aforementioned components of the computing device 1100 and Figure 11 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 11 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0250] The computing device 1100 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1100 can also be a mobile or stationary server.

[0251] The processor 1120 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described image processing method, skin lesion image processing method, or image processing model training method.

[0252] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solution of the image processing method, skin lesion image processing method, or image processing model training method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the image processing method, skin lesion image processing method, or image processing model training method described above.

[0253] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described image processing method, skin lesion image processing method, or image processing model training method.

[0254] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the image processing method, skin lesion image processing method, or image processing model training method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the image processing method, skin lesion image processing method, or image processing model training method described above.

[0255] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described image processing method, skin lesion image processing method, or image processing model training method.

[0256] The above is an illustrative scheme of a computer program according to this embodiment. It should be noted that the technical solution of this computer program belongs to the same concept as the technical solution of the image processing method, skin lesion image processing method, or image processing model training method described above. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the image processing method, skin lesion image processing method, or image processing model training method described above.

[0257] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0258] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0259] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0260] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0261] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An image processing method, comprising: The image processing task is received, wherein the image processing task carries an image to be detected corresponding to a target detection region, and the image processing task is used to detect whether there is an anomaly in the target detection region; The image to be detected is input into an image processing model to obtain the target detection result corresponding to the target detection region. The image processing model generates an initial prediction result based on the image to be detected, determines the correlation score of the initial prediction result based on the initial prediction result and the result relationship matrix, and determines the target detection result based on the correlation score. The result relationship matrix specifically refers to a matrix that identifies multiple known results. The result relationship matrix is ​​used to identify the correlation between multiple results so as to further filter the results in the initial prediction result, exclude mutually exclusive prediction results, and represent the correlation between any two results by numbers. The correlation includes: autocorrelation, similarity, unknown, and mutual exclusion.

2. The method as described in claim 1, wherein the image processing model comprises an embedding layer, a feature processing layer, a classification layer, and a comparison layer; The image to be detected is input into an image processing model to obtain the target detection result corresponding to the target detection region, including: The image to be detected is input into the embedding layer to obtain the embedded image features; The embedded image features are input into the feature processing layer to obtain image decoding features and anomaly feature information; The image decoding features and the anomaly feature information are input into the classification layer to obtain an initial prediction result; The initial prediction result is input into the comparison layer to obtain the target detection result, wherein the comparison layer includes a result relationship matrix.

3. The method as described in claim 2, wherein the image to be detected is input into the embedding layer to obtain embedded image features, comprising: Based on preset segmentation information, the image to be detected is segmented into multiple sub-images to be detected; The preset classifier and multiple sub-images to be detected are concatenated into the input image; The image to be input is input into the embedding layer to obtain embedded image features.

4. The method as described in claim 2, wherein the initial prediction result includes multiple initial prediction sub-results; The initial prediction result is input into the comparison layer to obtain the target detection result, including: Determine the initial prediction sub-result to be processed and at least one reference initial prediction sub-result, wherein the initial prediction sub-result to be processed is any one of a plurality of initial prediction sub-results; Based on the result relationship matrix, the initial prediction sub-result to be processed, and each reference initial prediction sub-result, calculate the correlation score corresponding to the initial prediction sub-result to be processed; The initial prediction results are sorted based on the correlation scores corresponding to each initial prediction sub-result to be processed, and the target detection results are determined.

5. The method as described in claim 4, wherein, based on the result relationship matrix, the initial prediction sub-result to be processed, and each reference initial prediction sub-result, the correlation score corresponding to the initial prediction sub-result to be processed is calculated, including: Based on the result relationship matrix, calculate the correlation sub-score between the initial prediction sub-result to be processed and each reference initial prediction sub-result; The correlation score corresponding to the initial prediction sub-result to be processed is determined based on the scores of each correlation sub-result.

6. The method of claim 1, wherein the image processing model is obtained through training via the following steps: Obtain the sample image and the corresponding sample detection result; The sample image is input into the image processing model to obtain the predicted detection results, predicted abnormal feature information and predicted non-abnormal feature information output by the image processing model; The model loss value is calculated based on the sample detection results, the predicted detection results, the predicted abnormal feature information, and the predicted non-abnormal feature information; The model parameters of the image processing model are adjusted based on the model loss value until the model training stops.

7. The method as described in claim 6, wherein the image processing model comprises an embedding layer, a feature processing layer, a classification layer, and a comparison layer; The sample image is input into an image processing model to obtain the predicted detection results, predicted abnormal feature information, and predicted non-abnormal feature information output by the image processing model, including: The sample image is input into the embedding layer to obtain the embedded sample image features; The embedded sample image features are input into the feature processing layer to obtain sample image decoding features, predicted abnormal feature information, and predicted non-abnormal feature information; The sample image decoding features and the predicted anomaly feature information are input into the classification layer to obtain the prediction detection result.

8. The method of claim 7, wherein the feature processing layer comprises at least one attention sublayer; The embedded sample image features are input into the feature processing layer to obtain sample image decoding features, predicted anomaly feature information, and predicted non-anomaly feature information, including: The embedded sample image features are sequentially input into each attention sub-layer to obtain the sample image decoding features and the attention feature matrix corresponding to each attention sub-layer; Initial association features corresponding to each attention feature matrix are extracted from each attention feature matrix, and target association features are determined based on each initial association feature. The initial association features represent the weight of abnormal targets in each sample sub-image, and the target association features include the distribution weight of abnormal targets in the sample images. Based on the target correlation characteristics, predict abnormal feature information and predict non-abnormal feature information are determined.

9. The method of claim 6, wherein calculating the model loss value based on the sample detection result, the predicted detection result, the predicted abnormal feature information, and the predicted non-abnormal feature information includes: Calculate the first loss value based on the sample detection results and the predicted detection results; Calculate the second loss value based on the predicted abnormal feature information and the predicted non-abnormal feature information; The model loss value is determined based on the first loss value and the second loss value.

10. A method for processing skin lesion images, comprising: Receive a skin lesion image processing task, wherein the skin lesion image processing task carries a skin lesion image to be detected corresponding to a target detection area, and the skin lesion image processing task is used to detect whether there is an abnormality in the target detection area; The image of the skin lesion to be detected is input into the skin lesion image processing model to obtain the target detection result corresponding to the target detection region. The skin lesion image processing model generates an initial prediction result based on the image of the skin lesion to be detected, and determines the correlation score of the initial prediction result based on the initial prediction result and the result relationship matrix. The target detection result is determined according to the correlation score. The result relationship matrix specifically refers to a matrix that identifies multiple known results. The result relationship matrix is ​​used to identify the correlation between multiple results so as to further filter the results in the initial prediction result, exclude mutually exclusive prediction results, and represent the correlation between any two results by numbers. The correlation includes: autocorrelation, similarity, unknown, and mutual exclusion.

11. A training method for an image processing model, applied to a cloud-based device, comprising: Obtain the sample image and the corresponding sample detection result; The sample image is input into the image processing model to obtain the predicted detection results, predicted abnormal feature information and predicted non-abnormal feature information output by the image processing model; The model loss value is calculated based on the sample detection results, the predicted detection results, the predicted abnormal feature information, and the predicted non-abnormal feature information; The model parameters of the image processing model are adjusted according to the model loss value until the model training stopping condition is met, thereby obtaining the model parameters of the image processing model. The model parameters of the image processing model are sent to the end-side device.

12. An image processing method, comprising: The system receives an image processing request sent by a user, wherein the image processing request includes an image processing task, the image processing task carries an image to be detected corresponding to a target detection region, and the image processing task is used to detect whether there is an anomaly in the target detection region; The image to be detected is input into an image processing model to obtain the target detection result corresponding to the target detection region. The image processing model generates an initial prediction result based on the image to be detected, determines the correlation score of the initial prediction result based on the initial prediction result and the result relationship matrix, and determines the target detection result based on the correlation score. The result relationship matrix specifically refers to a matrix that identifies multiple known results. The result relationship matrix is ​​used to identify the correlation between multiple results so as to further filter the results in the initial prediction result, exclude mutually exclusive prediction results, and represent the correlation between any two results by numbers. The correlation includes: autocorrelation, similarity, unknown, and mutual exclusion. Send the target detection result corresponding to the target detection area to the user.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images