Image processing method, model training method and medical image computer-aided diagnosis method

Abstract

The embodiment of the invention provides an image processing method, a model training method and a medical image computer-aided diagnosis method, and relates to the technical field of artificial intelligence, the image processing method comprises the following steps: in an image processing model, determining pruning parameters according to a text feature sequence and an image feature sequence, the image feature sequence is obtained by performing feature extraction on a to-be-processed image, and the text feature sequence is obtained by performing feature extraction on an image processing instruction of the to-be-processed image; pruning the image feature sequence according to the pruning parameters to obtain a plurality of reserved image features; and obtaining a processing result for responding to the image processing instruction based on the plurality of reserved image features and the text feature sequence. According to the technical scheme of the embodiment of the invention, adaptive pruning can be realized, and the accuracy of a model processing result is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method, a model training method, and a computer-aided diagnosis method for medical images, which can be applied to the field of intelligent question answering technology. Background Technology

[0002] In recent years, multimodal large language models have demonstrated powerful understanding and reasoning capabilities in medical question answering. However, when using multimodal large language models to process 3D medical images and corresponding image processing instructions, the large number of voxels in these images consumes significant storage and computational resources. To alleviate this problem, related techniques typically compress 3D medical images into a fixed number of visual tokens or prune the tokens using a fixed pruning threshold. However, both of these methods ignore the fact that different image processing instructions focus on different regions within the 3D medical image, thus reducing the accuracy of the model's processing results. Summary of the Invention

[0003] This application provides an image processing method, an image processing model training method, and a medical image computer-aided diagnosis method to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, embodiments of this application provide an image processing method applied to an image processing model. The method includes: determining pruning parameters based on a text feature sequence and an image feature sequence, wherein the image feature sequence is obtained by feature extraction from an image to be processed, and the text feature sequence is obtained by feature extraction from an image processing instruction for the image to be processed; pruning the image feature sequence according to the pruning parameters to obtain multiple retained image features; and obtaining a processing result for responding to the image processing instruction based on the multiple retained image features and the text feature sequence.

[0005] Secondly, embodiments of this application provide a training method for an image processing model, comprising: acquiring multiple training samples, each training sample including a training image and training instructions; training a network to be trained using the multiple training samples to obtain an image processing model, wherein the image processing model is used to implement any of the methods of embodiments of this application.

[0006] Thirdly, embodiments of this application provide a computer-aided diagnostic method for medical images, applied to an image processing model. The method includes: determining pruning parameters based on a text feature sequence and an image feature sequence, wherein the image feature sequence is obtained by feature extraction from a plain CT image of a target detection region, and the text feature sequence is obtained by feature extraction from a CT image processing instruction of the plain CT image; pruning the image feature sequence according to the pruning parameters to obtain multiple retained image features; and obtaining a processing result for responding to the CT image processing instruction based on the multiple retained image features and the text feature sequence.

[0007] Fourthly, embodiments of this application provide a computer-aided cancer diagnosis system, including a client and a server; The client is used to send a task processing request to the server, wherein the task processing request includes a plain CT image of the target detection area and CT image processing instructions for the plain CT image; The server is used to invoke the image processing model to obtain the processing result in response to the CT image processing command. The image processing model is used to implement the method of any one of the embodiments of this application.

[0008] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the methods of embodiments of this application when executing the computer program.

[0009] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any one of the embodiments of this application.

[0010] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method of any one of the embodiments of this application.

[0011] In the technical solution provided in this application embodiment, an image processing model determines pruning parameters based on the text feature sequence of the image processing instruction and the image feature sequence of the image to be processed, and then prunes the image feature sequence according to the determined pruning parameters. Therefore, for different image processing instructions, the corresponding pruning parameters can be dynamically determined, ensuring that the pruning parameters are compatible with the image processing instructions. Pruning is then performed based on these pruning parameters, resulting in multiple retained image features that have a strong correlation with the image processing instructions. This means that not only are useless features removed, but adaptive pruning based on the image processing instructions is also achieved. Consequently, the processing results obtained based on these multiple retained image features have high accuracy. Therefore, while improving the image processing speed, the accuracy of the model's processing results is also improved.

[0012] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0013] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments according to this application and should not be construed as limiting the scope of this application.

[0014] Figure 1A , Figure 1B and Figure 1C A schematic diagram of the architecture of the image processing system provided in an embodiment of this application is shown; Figure 2 A flowchart of the image processing method provided in an embodiment of this application is shown; Figure 3 A schematic diagram of the image processing model provided in an embodiment of this application is shown; Figure 4 A flowchart illustrating the training method of the image processing model provided in an embodiment of this application is shown; Figure 5 A flowchart of the medical image processing method provided in an embodiment of this application is shown; Figure 6 A block diagram of an image processing apparatus provided in an embodiment of this application is shown; Figure 7 A block diagram of a training apparatus for an image processing model provided in an embodiment of this application is shown; Figure 8 A block diagram of a medical image processing apparatus provided in an embodiment of this application is shown; Figure 9 A block diagram of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0015] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the concept or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0016] First, the terms used in the embodiments of this application are explained as follows: 3D medical imaging: Digital images that acquire three-dimensional (three-dimensional space) information of human organs, tissues, etc., through specific imaging techniques.

[0017] Multimodal Large Language Model (MLLM): A large language model that processes both visual information and textual instructions simultaneously, and can be used for tasks such as visual question answering.

[0018] 3D Vision Encoder: A 3D vision coding network based on the VisionTransformer, used to divide the input 3D image into 3D image blocks and map them into vector sequences.

[0019] Computed Tomography (CT): Also known as plain CT imaging, it uses precisely collimated X-ray beams, gamma rays, ultrasound, etc., along with highly sensitive detectors to scan a specific part of the human body one section after another. It features fast scanning time and clear images, and can be used to examine a variety of diseases.

[0020] Computer-aided diagnosis (CAD) refers to the use of imaging, medical image processing technology, and other possible physiological and biochemical methods, combined with computer analysis and calculation, to assist in the detection of lesions and improve the accuracy of diagnosis.

[0021] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and all of them fall within the protection scope of the embodiments of this application.

[0022] With the widespread application of multimodal large language models in medical vision question answering tasks, how to efficiently process 3D medical images has become a key issue. One approach to address this issue is fixed-length token compression, which compresses 3D medical images into a fixed number of tokens to reduce memory requirements. However, this method loses crucial details when processing complex image processing instructions and struggles to explain the contributions of different body parts.

[0023] In another approach, this critical issue can be addressed by using two-dimensional slicing or frame sampling, that is, by using two-dimensional slices of three-dimensional medical images or manually selecting a limited number of frames to replace complete three-dimensional medical image data. However, this disrupts the continuity of three-dimensional space, may miss important abnormal areas in the body, and manually selecting frames is prone to introducing bias.

[0024] In another implementation, this key issue can be addressed by uniform pruning or token merging. This involves using a uniform pruning rate and directly merging tokens based on attention or similarity to remove them. This approach ignores the fact that different image processing instructions often focus on different areas in 3D medical images, thus making it easier to remove relevant important tokens.

[0025] In another implementation, a learnable threshold can be introduced to address this key issue. This involves learning a pruning threshold during the training phase and then using the learned, fixed pruning threshold during the inference phase. However, this pruning threshold is not used during training, resulting in high training costs, and the fixed pruning threshold lacks adaptability to different image processing instructions.

[0026] In view of this, embodiments of this application provide an image processing method, an image processing model training method, and a medical image processing method, aiming to achieve adaptive pruning and improve the accuracy of model processing results, which will be described in detail below.

[0027] Figure 1A This application provides an embodiment of an image processing system, which is illustrated in the following schematic diagram. Figure 1A As shown, this architecture can include a client and a server. The client and server communicate with each other. The client can be a mobile phone, tablet, desktop computer, laptop, in-vehicle terminal, smart wearable device, or other terminal device. The server can be a physical server or a cloud server performing cloud computing. Figure 1A The example shown uses a desktop computer as the client and a physical server as the server.

[0028] Users can input an image to be processed and image processing instructions into the client. Upon receiving the user-inputted image and instructions, the client sends a task processing request containing both the image and instructions to the server. The server receives the task processing request and extracts features from both the image processing instructions and the image to be processed, obtaining text feature sequences and image feature sequences. The server then invokes an image processing model. Within the model, pruning parameters are determined based on the text and image feature sequences. The image feature sequences are then pruned according to these parameters, resulting in multiple retained image features. Based on these retained features and the text feature sequence, a processing result is obtained to respond to the image processing instructions. The server sends this processing result to the client, which then displays the received result. The image processing model is pre-trained and can be deployed on the server or other devices. For example, the image processing model is a multimodal large language model.

[0029] Figure 1B This application provides an embodiment of an image processing system, which is illustrated in the following schematic diagram: Figure 1B As shown, this architecture may include an image processing device. This image processing device can be a terminal device such as a mobile phone, tablet computer, desktop computer, laptop, in-vehicle terminal, or smart wearable device, or it can be a physical server or a cloud server performing cloud computing. Figure 1B The image processing device is illustrated using a desktop computer as an example.

[0030] An image processing device may include an input module and a display module. A user can input a task processing request containing the image to be processed and image processing instructions for that image through the input module. Upon receiving the user's task processing request, the image processing device extracts features from both the image processing instructions and the image to be processed, obtaining text feature sequences and image feature sequences. The image processing device then invokes an image processing model. Within the model, pruning parameters are determined based on the text and image feature sequences. The image feature sequences are then pruned according to these parameters, resulting in multiple retained image features. Based on these retained image features and the text feature sequence, a processing result is obtained to respond to the image processing instructions. The image processing device displays this processing result through the display module. The image processing model is pre-trained and can be deployed within the image processing device or on other devices. For example, the image processing model may be a multimodal large language model.

[0031] The aforementioned image processing system can be applied to image processing scenarios in any field. In some embodiments, the aforementioned image processing system is applied to image processing scenarios in the medical field, and correspondingly, such as... Figure 1C As shown, the user can input a plain CT image of the target detection area and CT image processing instructions for that image into the client. The client sends a task processing request to the server based on the received plain CT image and CT image processing instructions. The server receives the task processing request from the client and performs feature extraction on the CT image processing instructions and the plain CT image, respectively, to obtain text feature sequences and image feature sequences. The server invokes an image processing model to obtain the processing result used in response to the CT image processing instructions. The server sends this processing result to the client, and the client displays the received result. In this case, the image processing system can also be called a cancer computer-aided diagnostic system, where the processing result in response to the CT image processing instructions can characterize whether a tumor exists in the target detection area.

[0032] In the aforementioned image processing system, the corresponding pruning parameters are dynamically determined for different image processing instructions. This ensures that the pruning parameters are compatible with the image processing instructions, and pruning is performed based on these parameters. This results in a strong correlation between the multiple preserved image features obtained from the pruning and the image processing instructions; that is, not only are useless features removed, but adaptive pruning based on the image processing instructions is also achieved. Consequently, the processing results obtained based on these multiple preserved image features have high accuracy. Therefore, while improving the image processing speed, the accuracy of the model's processing results is also improved.

[0033] It should be noted that the application scenarios or examples provided in the embodiments of this application are for ease of understanding, and the embodiments of this application do not specifically limit the application of the technical solutions. In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0034] The technical solution of this application and how it solves the aforementioned technical problems are described in detail below with specific embodiments. The listed specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0035] Figure 2A flowchart of an image processing method according to an embodiment of this application is shown. Figure 2 The method shown can be applied to Figure 1A , Figure 1B and Figure 1C The image processing model shown is as follows. Figure 2 As shown, the method may include steps S201, S202 and S203.

[0036] Step S201: Determine pruning parameters based on text feature sequence and image feature sequence, wherein the image feature sequence is obtained by feature extraction of the image to be processed, and the text feature sequence is obtained by feature extraction of the image processing instructions of the image to be processed.

[0037] The image to be processed can be a three-dimensional image or a two-dimensional image. It can be an image from any scene, such as a medical image, a security image, or a vehicle driving image. In other words, the image processing method provided in this application embodiment can be applied to any scene, such as medicine, security, or intelligent driving; these will not be listed individually here. Image processing instructions are used to indicate the processing direction of the image to be processed. Image processing instructions can be in text or voice form. If the image processing instructions are in voice form, they can first be converted into text form, and then feature extraction can be performed on the text form to obtain a text feature sequence.

[0038] As an example, the image to be processed is a captured image of the liver; the image processing instruction could be to assess the morphology and density of the liver. As another example, the image to be processed is a security image; the image processing instruction could be to detect if someone has entered a restricted area. As yet another example, the image to be processed is an image captured during autonomous driving; the image processing instruction could be to analyze the safe distance to the vehicle in front, etc.

[0039] In one implementation, the image processing model may include multiple feature extraction layers and at least one pruning layer, which may be interspersed among the feature extraction layers. Each feature extraction layer performs attention computation based on relative position encoding on the input image feature sequence and text feature sequence to extract new image feature sequences and new text feature sequences. Each pruning layer determines pruning parameters based on the input text feature sequence and image feature sequence, and prunes the input image feature sequence according to the pruning parameters. That is, in any pruning layer of the image processing model, pruning parameters are determined based on the input text feature sequence and image feature sequence.

[0040] For example, such as Figure 3As shown, the image processing model includes eight feature extraction layers and one pruning layer. The eight feature extraction layers are denoted as Feature Extraction Layer 1, Feature Extraction Layer 2… Feature Extraction Layer 8, arranged from front to back. The pruning layer is positioned between Feature Extraction Layer 4 and Feature Extraction Layer 5. Accordingly, in the pruning layer, pruning parameters are determined based on the text feature sequence and image feature sequence extracted by Feature Extraction Layer 4. It should be noted that… Figure 3 This is for illustrative purposes only and is not intended to limit the scope of the image processing model. The model may also include other network layers, multiple pruning layers, and more or fewer feature extraction layers.

[0041] Step S202: Prune the image feature sequence according to the pruning parameters to obtain multiple preserved image features.

[0042] Since the image feature sequence may contain image features that are not related to the image processing instructions, in order to improve processing efficiency and reduce storage resource requirements, the image feature sequence can be pruned in the pruning layer according to the determined pruning parameters. That is, image features that are not related to the image processing instructions are pruned, or redundant features or useless features are pruned, so as to obtain multiple retained image features that are related to the image processing instructions.

[0043] Step S203: Based on multiple preserved image feature and text feature sequences, obtain the processing result for responding to image processing instructions.

[0044] After obtaining multiple preserved image features in the pruning layer of the image processing model, these multiple preserved image features and text feature sequences can be input into the next layer of the pruning layer so that feature extraction can be performed on the multiple preserved image features and text feature sequences through the next layer until the processing result of the image to be processed is obtained.

[0045] In the technical solution provided in this application embodiment, an image processing model determines pruning parameters based on the text feature sequence of the image processing instruction and the image feature sequence of the image to be processed, and then prunes the image feature sequence according to the determined pruning parameters. Thus, for different image processing instructions, the corresponding pruning parameters are dynamically determined, ensuring that the pruning parameters are compatible with the image processing instructions. Pruning is then performed based on these pruning parameters, resulting in multiple retained image features that have a strong correlation with the image processing instructions; that is, not only are useless features removed, but adaptive pruning is also achieved. Furthermore, the processing results obtained based on these multiple retained image features have high accuracy. Therefore, while improving the image processing speed, the accuracy of the model's processing results is also improved.

[0046] In one implementation, the image to be processed is a three-dimensional image. In order to balance spatial details and reduce computational costs, the preceding step S201 may include: dividing the image to be processed into multiple image blocks, merging the multiple image blocks in a specified dimension according to a merging step size to obtain multiple merged image blocks; extracting the initial features corresponding to the multiple merged image blocks respectively; dividing the image processing instructions into multiple text units, extracting the initial features corresponding to the multiple text units respectively; and inputting the initial features corresponding to the multiple merged image blocks and the initial features corresponding to the multiple text units respectively into the image processing model.

[0047] In one implementation, the image to be processed can be represented as Where H represents height, W represents width, and D represents depth. For example... Figure 3 As shown, after acquiring the image to be processed and the image processing instructions, the image processing device can invoke a 3D vision encoder. The embedding module of the 3D vision encoder divides the image to be processed into multiple non-overlapping image blocks (i.e., visual tokens) according to a preset size. To maintain depth resolution while reducing the number of image features, the embedding module of the 3D vision encoder also merges the multiple image blocks along the H and W dimensions with a step size S, resulting in multiple merged image blocks of number D. (H / S) (W / S) and embedding is performed on each merged image patch to obtain the embedding features of each merged image patch. The extraction module of the 3D vision encoder then performs feature extraction on the embedded features to obtain the initial features corresponding to each of the multiple merged image patches. The image processing device can also call a word segmenter to divide the image processing instructions into multiple text units (i.e., text tokens) and extract the initial features corresponding to each text unit. The initial features corresponding to the multiple merged image patches and the initial features corresponding to the multiple text units are concatenated to obtain concatenated initial features, which are then input into the image processing model.

[0048] The image processing device can simultaneously invoke the 3D visual encoder and the word segmenter, or it can invoke the 3D visual encoder first and then the word segmenter, or vice versa. The initial features corresponding to any merged image block and the initial features corresponding to any text unit can be vectors of the target dimension. The extraction module of the 3D visual encoder can include multiple attention blocks.

[0049] For example, the size of the image to be processed is = 8*8*3 (i.e., 8 pixels high, 8 pixels wide, and 3 slices deep), preset size is (That is, each image patch is 2 pixels high, 2 pixels wide, and 1 slice deep), with a merging step size of 2 (that is, merging every 2 image patches in the H and W dimensions). Therefore, for any slice, the partition is (8 / 2). (8 / 2) = 4 There are 16 two-dimensional image blocks (4 = 16), and a total of 48 image blocks (three slices). After merging, each slice becomes (4 / 2). (4 / 2) = 2 2 = 4 merged image blocks, and three slices make a total of 12 merged image blocks. As you can see, the number of image blocks is significantly reduced after merging.

[0050] Therefore, by merging multiple image patches of the image to be processed in the H and W dimensions, not only can the merged image patches integrate a wider range of contextual information, significantly reducing the number of image patches and lowering the computational cost, but also, since the D dimension remains unchanged, the model can still perceive information from three different slice levels, thus achieving a balance between spatial detail and computational cost.

[0051] In one implementation, to ensure the accuracy of pruning, the pruning parameters include a saliency index for any image feature in the image feature sequence. This saliency index characterizes the degree of correlation between the image feature and the image processing instruction. Accordingly, the aforementioned determination of pruning parameters based on the text feature sequence and the image feature sequence may include: for a target text feature in the text feature sequence, determining the weight corresponding to the target text feature based on the correlation between the target text feature and other text features, wherein the target text feature is any text feature in the text feature sequence, and the other text features are all text features in the text feature sequence other than the target text feature; and for any image feature in the image feature sequence, determining the saliency index of the image feature based on the weights corresponding to the image feature and each text feature in the text feature sequence.

[0052] To ensure that the saliency index is driven by image processing instructions, firstly, for any text feature in the text feature sequence (i.e., the target text feature), its weight is determined based on its correlation with other text features in the sequence. This weight characterizes the importance of the text feature in the text feature sequence. After obtaining the weights of each text feature in the text sequence, to evaluate the correlation between each image feature and the image processing instructions, for any image feature in the image feature sequence, a saliency index for the image feature is determined based on the image feature and the weights of each text feature in the text feature sequence.

[0053] For example, such as Figure 3As shown, the pruning layer may include an instruction-aware saliency evaluation (ISE) module. Accordingly, the determination of the aforementioned saliency indicators can be performed through this ISE module. The specific network structure of the ISE module can be configured as needed in practical applications.

[0054] Therefore, by first determining the weight corresponding to any text feature in the text feature sequence, and then determining the saliency index of each image feature based on the weight, it is ensured that the saliency index is driven by the image processing instructions, that is, it is ensured that the determined pruning parameters are compatible with the image processing instructions.

[0055] In one implementation, determining the weight corresponding to the target text feature based on the correlation between the target text feature and other text features may include: determining the attention scores between the target text feature and other text features respectively; aggregating the attention scores to obtain the aggregated score corresponding to the target text feature; and performing a first transformation on the aggregated score to obtain the weight corresponding to the target text feature.

[0056] In one implementation, attention can be calculated on the target text feature and any other text feature to obtain the inner product between the target text feature and each other text feature. This inner product is then pre-processed with respect to the dimension of the attention head in the attention calculation to obtain an attention score. After obtaining the attention scores between the target text feature and each other text feature, the attention scores can be summed to obtain the aggregate score corresponding to the target text feature. This aggregate score is then normalized using a normalization function (e.g., the Softmax function) to obtain the weight corresponding to the target text feature.

[0057] Since each text feature is a two-dimensional vector, to avoid negative attention scores, in one implementation, after obtaining any attention score, the method may further include: comparing the attention score with zero, and updating the attention score to zero if the attention score is less than zero. That is, the weight corresponding to the target text feature can be expressed as: .

[0058] in, Represents a text feature sequence. Represents the target text features (i.e., any text feature). Indicates other text features, This refers to the dimension of the attention head in attention computation. This represents the inner product operation. This indicates taking the maximum value.

[0059] Therefore, by determining the attention score between the target text feature and other text features in the text feature sequence, and determining the weight of the target text feature based on the attention score, the importance of the target text feature can be characterized by the weight, providing a basis for subsequently determining the saliency index of image features driven by image processing instructions.

[0060] In one implementation, the aforementioned determination of the saliency index of image features based on the weights corresponding to each text feature in the image feature and text feature sequence may include: determining the similarity between the image feature and the text feature for any text feature in the text feature sequence; performing a weighted summation of the corresponding similarities based on the weights corresponding to each text feature in the text feature sequence to obtain the weighted similarity of the image features; and performing a second transformation on the weighted similarity to obtain the saliency index of the image features.

[0061] For example, for any image feature in the image feature sequence, attention can be calculated between the image feature and each text feature in the text feature sequence to obtain the inner product between the image feature and each text feature. The inner product and the dimension of the attention head in the attention calculation are then pre-processed to obtain the similarity between the image feature and the corresponding text feature. Based on the weights corresponding to each text feature in the text feature sequence, each similarity is weighted and summed to obtain the weighted similarity of the image feature. This weighted similarity is then min-max normalized to obtain the significance index of the image feature.

[0062] In other words, the significance index of any image feature can be expressed as: .

[0063] in, In the image feature sequence, the first... Image features Significance indicators This indicates that min-max normalization is being performed. Representing text feature sequences Any text feature in, Representing text features The corresponding weights.

[0064] It should be noted that the similarity between image features and text features is not limited to the above determination method. Cosine distance, Euclidean distance, etc. between image features and text features can also be used as the similarity between image features and text features.

[0065] Therefore, for any image feature, by determining the similarity between the image feature and each text feature, the degree of attention paid to the image feature by the corresponding text feature can be represented by the similarity. Then, the similarity is weighted and normalized to serve as the significance index of the image feature. This significance index can accurately measure the degree of association between the image feature and the image processing instructions, thus providing a guarantee for accurate pruning in the future.

[0066] To avoid the problem of poor adaptability of fixed thresholds to different image processing instructions in related technologies, in one embodiment, after determining the saliency index of each image feature based on instruction-driven determination, an index threshold can be predicted based on each saliency index, and each saliency index and the index threshold can be determined as pruning parameters. That is, the aforementioned determination of pruning parameters based on text feature sequences and image feature sequences can include: determining the distribution features, scale features, and compression features corresponding to the saliency index of each image feature in the image feature sequence, and predicting the index threshold based on the distribution features, scale features, and compression features. The distribution features characterize the overall distribution of each saliency index, the scale features characterize the original range corresponding to each saliency index, and the compression features characterize the concentration of important image features.

[0067] In one implementation, at least one of the following significance indicators can be determined as distribution features: mean (characterizing the central tendency of the distribution), standard deviation (characterizing the dispersion of the distribution), maximum and minimum significance indicators (characterizing the boundary range of the distribution), and quartiles (characterizing the spread and skewness of the distribution). For the weighted similarity corresponding to the significance indicator of any image feature in the image feature sequence, a logarithmic transformation is performed on the weighted similarity to obtain transformed data. At least one of the following transformation data is determined as a scale feature: mean (characterizing the central position of the transformed data on a logarithmic scale), standard deviation (characterizing the fluctuation of the transformed data on a logarithmic scale), maximum and minimum values ​​(characterizing the boundary range of the transformed data). Furthermore, for the significance indicator of any image feature in the image feature sequence, nonlinear compression is performed on the significance indicator to obtain compressed data. At least one of the following compression data is determined as a compression feature: mean, maximum value, etc. The distribution features, scale features, and compressed features are concatenated, and a threshold indicator is predicted based on the concatenated features.

[0068] The logarithmic transformation can be expressed as: ,in, The significance index of the j-th image feature in the image feature sequence. The corresponding weighted similarity, As a preset constant, This represents taking the logarithm. Nonlinear compression can be expressed as... ,in, The significance index represents the j-th image feature in the image feature sequence. Compressed data obtained by nonlinear compression Represents a symbolic function. The significance index of the j-th image feature. The absolute value of . This nonlinear compression can stretch low-value regions and compress high-value regions, thus being more sensitive to changes in the high-value tail. Let the concatenated feature be denoted as z, and the predicted index threshold can be expressed as . , [0,1], where, This represents the Sigmoid function. This represents the GELU activation function. , , and These are parameters obtained through model training.

[0069] In one implementation, such as Figure 3 As shown, the pruning layer includes an instance-aware threshold prediction (ITP) module. Accordingly, the above distribution features, scale features, and compression features can be determined through this ITP module, and the index threshold can be predicted.

[0070] By determining the distribution characteristics, the overall distribution of each saliency index can be characterized by its shape and dispersion. Since max-min normalization of the weighted similarity loses original scale information, a logarithmic transformation of the weighted similarity and determination of scale characteristics allow for the representation of the original range of each saliency index using these coarse features. Because image features with high saliency are more important during pruning, a compressed feature can be determined to represent the concentration of important image features. Therefore, this multi-angle complementary feature extraction method ensures that the predicted thresholds are driven by the image to be processed and the image processing instructions, making them highly adaptive and intelligently balancing computational efficiency and accuracy under different images and processing instructions.

[0071] After obtaining the threshold, the image feature sequence can be pruned according to pruning parameters, which include the threshold and the saliency index of any image feature in the image feature sequence. That is, the aforementioned pruning of the image feature sequence according to the pruning parameters can include: for any image feature in the image feature sequence, if the saliency index of the image feature is less than the threshold, the image feature is pruned.

[0072] For example, for any image feature in the image feature sequence, the saliency index of the image feature can be compared with an index threshold, and if the saliency index is less than the index threshold, the image feature can be removed.

[0073] Since the index threshold is determined based on the image processing instructions and the image to be processed, pruning is performed by comparing the saliency index of the image features with the index threshold, thus ensuring the accuracy of pruning and consequently the accuracy of the image processing results.

[0074] Corresponding to the image processing method provided in the embodiments of this application, the embodiments of this application also provide a training method for an image processing model, which can be applied to a training device, such as... Figure 4 As shown, the method includes the following steps S401 and S402.

[0075] Step S401: Obtain multiple training samples, each training sample including a training image and training instructions; Step S402: Train the network to be trained using multiple training samples to obtain an image processing model.

[0076] The network to be trained may include a 3D visual encoder network and a multimodal network. The 3D visual encoder network can be obtained by adding the aforementioned embedding module to an existing pre-trained 2D visual encoder, and the multimodal network can be obtained by adding at least one pruned layer to an existing multimodal language model. Unlike the three-stage model training method in related technologies (i.e., first pre-training the visual encoder, then performing visual-text alignment training, and finally fine-tuning the language model), the training of the network to be trained in this application can be divided into two stages. In the first stage, all modules in the multimodal network and the 3D visual encoder network except for the embedding module can be frozen, and only the embedding module is fine-tuned. In the second stage, all modules are unfrozen, the multimodal network is fine-tuned, and the multimodal network that meets the training termination condition is determined as the image processing model. The fine-tuning process of the embedding module can refer to related technologies, and will not be elaborated here. The process of fine-tuning the multimodal network is described in detail below.

[0077] In one implementation, the aforementioned method of training the network to be trained using multiple training samples to obtain an image processing model may include: determining pruning parameters for the training samples based on the text feature sequence and image feature sequence corresponding to any training sample; generating surrogate masks for each image feature in the image feature sequence based on the pruning parameters; pruning the image feature sequence based on the pruning parameters to obtain multiple retained image features; and updating the parameters of the network to be trained based on the multiple retained image features and the surrogate masks to obtain the image processing model.

[0078] The process of determining the pruning parameters can be found in the previous description and will not be repeated here. Since the gradient of the pruned image features is zero during backpropagation after pruning the image feature sequence, the model cannot learn whether the pruning strategy for those features is correct. Therefore, in one implementation, after determining the pruning parameters for the training samples, a surrogate mask for each image feature in the image feature sequence is generated based on these parameters. The parameters of the network to be trained are then updated based on multiple retained image features and the surrogate masks.

[0079] Therefore, by determining corresponding pruning parameters and performing pruning for each training sample, the model can learn the appropriate pruning strategy for different training samples, thereby improving the accuracy of the processing results during the inference stage. By generating a surrogate mask, the parameters of the network to be trained can be effectively updated.

[0080] As mentioned above, the pruning parameters include an index threshold and a salience index of any image feature in the image feature sequence. Accordingly, the aforementioned generation of surrogate masks for each image feature in the image feature sequence based on the pruning parameters may include: for any image feature in the image feature sequence, determining the retention probability of the image feature based on the salience index and the index threshold; and performing a third transformation on the retention probability to obtain a surrogate mask for high-value image features.

[0081] In one implementation, for any image feature in the image feature sequence, the saliency index and index threshold of the image feature can be processed using the Sigmoid function to obtain the retention probability of the image feature. A binary mask of the retention probability is obtained, and the binary mask and the retention probability are subjected to preset processing to obtain a surrogate mask for the image feature.

[0082] In one implementation, the retention probability can be expressed as: ,in, This represents the retention probability of the j-th image feature in the image feature sequence. This represents the Sigmoid function. Indicates the threshold value of the indicator. The significance index represents the j-th image feature in the image feature sequence. This indicates the slope that controls the Sigmoid function.

[0083] In one implementation, obtaining the binary mask representing the retention probability may include: if the retention probability is greater than a preset data value, determining the binary mask as first data; otherwise, determining the binary mask as second data. Correspondingly, during training, pruning the image feature sequence according to pruning parameters may include: for any image feature, pruning the image feature when its binary mask is the second data value. For example, the preset data is 0.5, the first data is 1, and the second data is 0. That is, the binary mask can be represented as... That is, when the retention probability of the j-th image feature is greater than 0.5, its binary mask... The value is 1.

[0084] Furthermore, the proxy mask can be represented as M_j=sg ( -sg )+ , where sg() represents the stopping gradient operator, and M_j is the surrogate mask for the j-th image feature.

[0085] Therefore, by determining the retention probability of each image feature, obtaining the binary mask of the retention probability, and performing pruning based on the binary mask and determining the surrogate mask, not only is the consistency of the pruning operation in the training and inference processes guaranteed, but also the problem that discrete pruning cannot optimize the model is solved because the parameters are updated based on the surrogate mask of each image feature.

[0086] In one implementation, the parameters of the network to be trained may include parameters of a first network layer and parameters of a second network layer. The first network layer is used for pruning, and the second network layer consists of all network layers in the network to be trained except for the first network layer. Accordingly, the aforementioned updating of the parameters of the network to be trained based on multiple preserved image features and proxy masks may include: Scattering the surrogate mask yields a gradient tensor, which has the same dimension as the image feature sequence. Based on the position of the preserved image feature in the image feature sequence, the gradient of the preserved image feature is filled into the gradient tensor, resulting in a new gradient tensor. For any image feature in the image feature sequence, a surrogate gradient corresponding to that image feature is generated based on the surrogate mask and the corresponding initial feature, where the initial feature is the feature input to the network to be trained. The parameters of the second network layer are updated based on the new gradient tensor. The parameters of the first network layer are updated based on the surrogate gradients corresponding to each image feature in the image feature sequence.

[0087] For example, during forward propagation, pruning parameters can be determined for any training sample in the first network layer, and the corresponding image feature sequence can be pruned according to the pruning parameters to obtain multiple preserved image features. These multiple preserved image features are then input into the next layer of the first network layer until the output of the network to be trained is obtained, and the loss corresponding to the training sample is determined based on the output. During backpropagation, the gradients of the trainable parameters in any network layer are determined according to the loss function and propagated forward. After obtaining each gradient, the parameters of the network to be trained are updated uniformly according to each gradient.

[0088] In one implementation, determining the loss corresponding to the training sample based on the output result may include: determining a cross-entropy loss based on the output result and the standard output result carried in the training sample; determining a constraint loss based on the retention probability of each image feature in the training sample; determining a robust regularization loss based on the training image and the perturbed image of the training image in the training sample, wherein the perturbed image refers to an image that has been perturbed (e.g., occluded, scrambled, etc.) and has lost some features; determining a flip regularization loss based on the training image, the standard output result, and a randomly generated flip mask in the training sample, wherein the flip mask is an erroneous binary mask. The cross-entropy loss, constraint loss, robust regularization loss, and flip regularization loss are added together to obtain the loss corresponding to the training sample.

[0089] In one implementation, the constraint loss can be expressed as ,in, Indicates constraint loss, The layer weights of the first network layer. This represents the average retention probability of each image feature corresponding to the training samples. This represents the upper limit of the preset average retention probability. This represents the lower limit of the preset average retention probability.

[0090] Robust regularization loss can be expressed as ,in, Indicates the robust regularization loss. Indicates the expected value. Represents the training images in the training samples The perturbation image, This represents a complete probability distribution, which includes the probability distribution of the network to be trained on a given perturbed image. In the case of "yes" and "no", the probability distribution includes the probability of "yes" being 0.6 and the probability of "no" being 0.4. Let v be the information entropy, and v represent the prediction results traversed through the aforementioned complete probability distribution. This represents the probability of predicting outcome v within the complete probability distribution.

[0091] The flip regularization loss is expressed as ,in, Indicates the inversion regularization loss. Indicates the expected value. To flip the mask, This represents the training images in the training samples. This represents the standard output result in the training samples. As a preset constant, Indicates that given training images and flip mask In the case of [condition], the probability that the network to be trained will obtain the standard output result.

[0092] Since there may be excessive retention or pruning of image features without additional constraints, this application introduces the aforementioned constraint loss to avoid this problem. When the average retention probability is too high or too low, the constraint loss will penalize it and pull it back to the stable range, thus avoiding excessive imbalance in the pruning ratio.

[0093] To mitigate the risk of language hallucinations—that is, when the input is an occluded or scrambled image—the model should exhibit greater instability rather than relying on prior textual outputs. Therefore, this application introduces a robust regularization loss to encourage the network to maintain a high-entropy distribution in its responses to missing features, thereby suppressing language-driven hallucinations. For example, to balance training efficiency, the robust regularization loss can be determined by randomly selecting a subset of training samples, rather than determining it for every single training sample.

[0094] To avoid model dependence on specific mask patterns, this application introduces a flip regularization loss. This loss flips the binary mask with a certain probability during training, preserving image features that should be discarded and discarding those that should be preserved. If the model outputs a standard result under the flipped mask, this term increases the penalty, prompting the model to rely on the true preserved pattern. Furthermore, by determining the loss based on the above factors and adjusting the model parameters, the accuracy of the model can be guaranteed.

[0095] Furthermore, since the pruned image features are discrete, the gradients of the pruned image features cannot be obtained in the first network layer during backpropagation. Therefore, in one implementation, during backpropagation, a zero-dimensional gradient tensor with the same dimension as the image feature sequence can be created in the first network layer. For any retained image feature, based on its position in the image feature sequence, the value at the corresponding position in the gradient tensor is updated to represent the gradient of the retained image feature. After the gradients of all retained image features are filled into the gradient tensor, a new gradient tensor is obtained. Since the gradients of the pruned image features remain zero in this new gradient tensor, the parameters of the second network layer are updated based on this new gradient tensor. Additionally, a proxy gradient for each image feature is generated based on its proxy mask, and the parameters of the first network layer are updated based on this proxy gradient.

[0096] Since the gradients of preserved image features are discrete and non-differentiable during backpropagation, this non-differentiability issue is resolved by scattering the surrogate mask to obtain a new gradient tensor. By generating surrogate gradients, the pruned image features also acquire gradients, allowing the model to learn the accuracy of the pruning strategy and thus effectively update the parameters of the first network layer.

[0097] In one implementation, the aforementioned generation of a surrogate gradient corresponding to an image feature based on a surrogate mask and an initial feature corresponding to the image feature may include: determining the contribution of the image feature based on the initial feature corresponding to the image feature, wherein the contribution is used to characterize the impact of deleting the image feature on model performance; determining the orientation factor corresponding to the image feature based on the contribution and the surrogate mask, wherein the orientation factor is used to indicate whether the contribution of the image feature to completing the training instruction is positive or negative; determining the magnitude factor corresponding to the image feature based on the initial feature and the gradient corresponding to the image feature in the updated gradient tensor, wherein the magnitude factor is used to characterize the correlation between the image feature and the training instruction; and determining the surrogate gradient of the image feature based on the orientation factor and the magnitude factor.

[0098] In one implementation, determining the contribution of an image feature based on its corresponding initial feature may include: for any image feature, using a first-order Taylor expansion to determine its contribution. This involves determining the gradient of the loss function on the initial feature of that image feature, and performing an inner product operation between this gradient and the initial feature to obtain the contribution of that image feature. In other words, the contribution can be expressed as... ,in, This represents the contribution of the j-th image feature in the image feature sequence. This represents the initial feature of the j-th image feature (i.e., the feature output by the 3D visual encoder). This indicates that the loss function is applied to the initial features. The gradient.

[0099] In one implementation, determining the orientation factor corresponding to an image feature based on its contribution and a surrogate mask may include: performing a Hadamard product operation on the contribution and surrogate mask of any retained image feature to obtain the operation result; determining the mean and standard deviation of the operation results for each retained image feature; performing a fourth transformation on the mean and standard deviation to obtain the transformation result; and mapping the transformation result to obtain the orientation factor of the image feature.

[0100] The process of the fourth transformation can be represented as follows: .

[0101] in, This represents the transformation result corresponding to the j-th image feature. This represents the contribution of the j-th image feature. This represents the average of the results of each operation. The standard deviation of each operation result is represented. For non-zero parameters, This indicates taking the maximum value. The mapping process of the transformation result can be represented as follows: ,in, Indicates the direction factor. Let be a smooth function, and sg() denote the stopping gradient operator. When <0, the contribution of the corresponding image features to completing the training instructions is positive. When the value is greater than 0, the contribution of the corresponding image features to completing the training instructions is negative.

[0102] In one implementation, as described above, any image feature is a vector of the target dimension. Since the gradient tensor has the same dimension as the image feature sequence, each gradient in the gradient tensor is also a vector of the target dimension. Accordingly, determining the magnitude factor corresponding to the image feature based on the initial features of the image feature and the gradient corresponding to the image feature in the updated gradient tensor may include: determining the coupling strength corresponding to the image feature based on the initial features of the image feature and the gradient corresponding to the image feature in the updated gradient tensor; and performing a fifth transformation on the coupling strength to obtain the magnitude factor corresponding to the image feature.

[0103] The coupling strength can be expressed as: , This represents the initial coupling strength between the j-th image feature and the gradient. The total dimension representing the image features (i.e., the target dimension). The first representing the image features One dimension, The initial feature representing the j-th image feature The Middle Values ​​of each dimension This represents the gradient of the j-th image feature in the new gradient tensor. The values ​​of each dimension.

[0104] The fifth transformation can be expressed as .

[0105] in, Let the magnitude factor of the j-th image feature be denoted as . () represents the clipping function. Indicates the weights of the first network layer. This represents the mean of the magnitude factors for each preserved image feature. It is a constant. It is the lower bound of the clipping function. It is the upper bound of the clipping function.

[0106] In one implementation, the proxy gradient can be expressed as: ,in, Represents the surrogate gradient of the j-th image feature. Indicates the global scaling factor. The direction factor represents the j-th image feature. The magnitude factor of the j-th image feature. This represents the retention probability of the j-th image feature. For a very small positive number (e.g.) ).

[0107] Therefore, by determining the orientation factor and magnitude factor of each image feature, the orientation factor can characterize whether the contribution of the corresponding image feature to completing the training instruction is positive or negative, and the magnitude factor can characterize the degree of correlation between the corresponding image feature and the training instruction. Thus, the surrogate gradient can be determined based on the orientation factor and magnitude factor. The quality of the decision to retain or prune the corresponding image feature can be evaluated through the surrogate gradient, thereby improving the model accuracy.

[0108] Based on the same technical concept as any of the foregoing embodiments, this application also provides a computer-aided diagnostic method for medical images. Figure 5 A flowchart of a computer-aided diagnostic method for medical images provided in an embodiment of this application is shown. Figure 5 The method shown can be used by Figure 1A , Figure 1B and Figure 1C The image processing model shown is executed as follows: Figure 5 As shown, the method may include steps S501, S502, and S503: Step S501: Determine pruning parameters based on text feature sequence and image feature sequence, wherein the image feature sequence is obtained by extracting features from the plain CT image of the target detection area, and the text feature sequence is obtained by extracting features from the CT image processing instructions of the plain CT image.

[0109] Step S502: Prune the image feature sequence according to the pruning parameters to obtain multiple preserved image features.

[0110] Step S503: Based on multiple preserved image feature and text feature sequences, obtain the processing results for responding to CT image processing commands.

[0111] The plain CT images can be two-dimensional or three-dimensional. The medical image computer-aided diagnostic method provided in this application can be applied to various medical tasks, including: spatial location of anatomical structures or abnormalities, lesion presence detection, quantitative image parameter extraction, lesion temporal evolution analysis, organ or lesion segmentation, abnormality classification and auxiliary diagnosis, and generation of structured radiological reports. Specifically, the spatial location of anatomical structures or abnormalities task involves accurately identifying and determining the location of key anatomical landmarks or lesions in the target detection area, such as detecting the location of pleural effusion. The lesion presence detection task involves screening and identifying the presence of specific diseases or abnormalities, such as screening for the presence of renal cysts. The quantitative image parameter extraction task involves obtaining quantifiable objective measurement data from images, such as extracting the average HU (Hounsfield Unit, used to quantify tissue density values ​​in plain CT images) value of the pancreas. The lesion temporal evolution analysis task involves evaluating the dynamic changes of lesions by comparing images from different periods, such as analyzing whether new ground-glass opacities in the lungs are developing or being absorbed. Organ or lesion segmentation tasks involve precisely outlining specific regions at the pixel level, such as fully automated processing of liver images to accurately segment the boundaries of liver parenchyma, tumors, and major vascular structures. Abnormality classification and assisted diagnosis tasks involve qualitatively classifying identified abnormalities to aid clinical diagnosis, such as further analyzing the morphological characteristics (e.g., lobulation, spiculation) of solid nodules detected in plain CT images of the lungs and classifying them as either highly benign or of high malignancy risk. The radiological structured report generation task involves automatically generating structured reports conforming to clinical standards based on plain CT images, such as automatically generating a structured report containing sections on "examination techniques, contrast agents, imaging findings, and impression diagnosis" after analyzing plain CT images of the brain, detailing the lesion's location, size, signal characteristics, and relationship with surrounding tissues. Correspondingly, CT image processing commands are natural language query commands or structured analysis commands corresponding to the aforementioned tasks.

[0112] As an example, the target detection area is the pleural cavity, and the CT image processing instruction is "Where is the pleural effusion more obvious?". The image processing model identifies the distribution of pleural effusion based on multiple sequences of preserved image and text features, obtains a first probability of effusion on the right side and a second probability of effusion on the left side, and determines the side with the highest probability (e.g., the right side) as the processing result to respond to the CT image processing instruction, and outputs this processing result "right side".

[0113] As another example, the target detection area is the kidney, and the CT image processing command is "Is there a cyst in the kidney?". The image processing model detects the presence of a cyst in the kidney based on multiple sequences of preserved image features and text features, obtaining a third probability of the presence of a cyst and a fourth probability of the absence of a cyst. Based on the maximum probability (e.g., the third probability), the processing result used to respond to the CT image processing command is determined to be "yes", and the processing result "yes" is output.

[0114] As another example, the target detection region is the pancreas, and the CT image processing instruction is "What is the average HU value of the pancreas? Options: A: 13.6; B: 16.7; C: 15.2; D: 9.9". The image processing model detects the density of tissue in the pancreas based on multiple sequences of preserved image and text features, and determines option A as the processing result to be used in response to the CT image processing instruction, outputting the processing result "A: 13.6".

[0115] As another example, the target detection area is the lungs, and the CT image processing instruction is "Previously, the lung lesion appeared as [1, 0, 0, 1, 1, 1, 1] (Note: 0 represents "none", 1 represents "present"). How has the lesion behaved in past and present CT scans? Compared to the historical sequence, which of the following options best describes the lung lesion now? Options: A. Refractory lesion (persistent or recurrent, now present) B. Regressive lesion (previously present or recurrent, now disappeared) C. New lesion (previously absent, now present) D. No abnormality (always absent)". The image processing model, based on multiple sequences of preserved image and text features, determines that a lesion that existed previously and currently exists meets the definition of "refractory lesion". Option A is selected as the processing result in response to the CT image processing instruction, and the detection result "A. Refractory lesion (persistent or recurrent, now present)" is output.

[0116] The specific implementation methods of steps S501 to S503 are the same as those of steps S201 to S203 mentioned above, and can be referred to the relevant descriptions above, which will not be repeated here.

[0117] Because pruning parameters can be dynamically determined for different medical image processing instructions, ensuring that these parameters are compatible with the instructions, pruning is performed based on these parameters. This results in multiple preserved image features having a strong correlation with the medical image processing instructions; that is, not only are useless features removed, but adaptive pruning is also achieved. Consequently, the processing results obtained based on these preserved image features have high accuracy. Therefore, while improving the medical image processing speed, the accuracy of the model's processing results is also improved.

[0118] Corresponding to the image processing method provided in the embodiments of this application, the embodiments of this application also provide an image processing apparatus, which can be applied to... Figure 1A , Figure 1B and Figure 1C The image processing model shown is as follows: Figure 6 As shown, the device includes: a determining module 601, used to determine pruning parameters based on a text feature sequence and an image feature sequence, wherein the image feature sequence is obtained by feature extraction from the image to be processed, and the text feature sequence is obtained by feature extraction from the image processing instructions of the image to be processed; a pruning module 602, used to prune the image feature sequence according to the pruning parameters to obtain multiple retained image features; and an acquiring module 603, used to obtain a processing result for responding to the image processing instructions based on the multiple retained image features and the text feature sequence.

[0119] In one embodiment, the pruning parameters include an index threshold and a saliency index of any image feature in the image feature sequence. The saliency index is used to characterize the degree of association between the image feature and the image processing instruction. Specifically, the pruning module 602 is used to prune any image feature in the image feature sequence when the saliency index of the image feature is less than the index threshold.

[0120] In one embodiment, the determining module 601 is specifically configured to: for a target text feature in the text feature sequence, determine the weight corresponding to the target text feature based on the correlation between the target text feature and other text features, wherein the target text feature is any text feature in the text feature sequence, and the other text features are all text features in the text feature sequence other than the target text feature; and for any image feature in the image feature sequence, determine the saliency index of the image feature based on the weights corresponding to the image feature and each text feature in the text feature sequence.

[0121] In one embodiment, the determining module 601 is further specifically configured to: determine the attention scores between the target text feature and the other text features respectively; aggregate the attention scores to obtain the aggregate score corresponding to the target text feature; and perform a first transformation on the aggregate score to obtain the weight corresponding to the target text feature.

[0122] In one embodiment, the determining module 601 is further specifically used to: determine the similarity between the image feature and the text feature for any text feature in the text feature sequence; and perform a weighted summation of the corresponding similarities according to the weights of each text feature in the text feature sequence to obtain the weighted similarity of the image feature. A second transformation is performed on the weighted similarity to obtain the saliency index of the image features.

[0123] In one embodiment, the determining module 601 is further specifically used to: determine the distribution features, scale features, and compression features corresponding to the saliency indicators of each image feature in the image feature sequence, wherein the distribution features are used to characterize the overall distribution of each saliency indicator, the scale features are used to characterize the original range corresponding to each saliency indicator, and the compression features are used to characterize the concentration of important image features; and predict the indicator threshold based on the distribution features, the scale features, and the compression features.

[0124] In one embodiment, the image to be processed is a three-dimensional image, and the method further includes a preprocessing module for dividing the image to be processed into multiple image blocks, merging the multiple image blocks in a specified dimension according to a merging step size to obtain multiple merged image blocks; extracting initial features corresponding to the multiple merged image blocks respectively; dividing the image processing instruction into multiple text units, extracting initial features corresponding to the multiple text units respectively; and inputting the initial features corresponding to the multiple merged image blocks and the initial features corresponding to the multiple text units respectively into the image processing model.

[0125] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0126] Corresponding to the image processing model training method provided in the embodiments of this application, the embodiments of this application also provide an image processing model training apparatus, which can be applied to training devices, such as... Figure 7 As shown, the device includes: an acquisition module 701, used to acquire multiple training samples, each training sample including a training image and training instructions; and a training module 702, used to train the network to be trained using the multiple training samples to obtain an image processing model, wherein the image processing model is used to implement the image processing method provided in any embodiment.

[0127] In one implementation, the training module 702 is specifically configured to: determine the pruning parameters of the training sample based on the text feature sequence and image feature sequence corresponding to any training sample; generate a surrogate mask for each image feature in the image feature sequence based on the pruning parameters; prune the image feature sequence based on the pruning parameters to obtain multiple retained image features; and update the parameters of the network to be trained based on the multiple retained image features and the surrogate mask to obtain the image processing model.

[0128] In one embodiment, the pruning parameters include an index threshold and a saliency index of any image feature in the image feature sequence. The training module 702 is further specifically used to: determine the retention probability of any image feature in the image feature sequence based on the saliency index and the index threshold of the image feature; and perform a third transformation on the retention probability to obtain a surrogate mask of the image feature.

[0129] In one implementation, the parameters of the network to be trained include parameters of a first network layer and parameters of a second network layer. The first network layer is used for pruning, and the second network layer consists of all network layers in the network to be trained except for the first network layer. The training module 702 is further specifically configured to: scatter the surrogate mask to obtain a gradient tensor, the gradient tensor having the same dimension as the image feature sequence; fill the gradient of the retained image feature into the gradient tensor according to the position of the retained image feature in the image feature sequence to obtain a new gradient tensor; for any image feature in the image feature sequence, generate a surrogate gradient corresponding to the image feature according to the surrogate mask of the image feature and the initial feature corresponding to the image feature, wherein the initial feature is the feature input to the network to be trained; update the parameters of the second network layer according to the new gradient tensor; and update the parameters of the first network layer according to the surrogate gradients corresponding to each image feature in the image feature sequence.

[0130] In one implementation, the training module 702 is further specifically configured to: determine the contribution of the image feature based on the initial feature corresponding to the image feature, wherein the contribution is used to characterize the impact of deleting the image feature on model performance; determine the direction factor corresponding to the image feature based on the contribution and the surrogate mask of the image feature, wherein the direction factor is used to indicate whether the contribution of the image feature to completing the training instruction is positive or negative; determine the magnitude factor corresponding to the image feature based on the initial feature of the image feature and the gradient corresponding to the image feature in the updated gradient tensor, wherein the magnitude factor is used to characterize the correlation between the image feature and the training instruction; and determine the surrogate gradient of the image feature based on the direction factor and the magnitude factor.

[0131] Corresponding to the medical image processing method provided in the embodiments of this application, the embodiments of this application also provide a medical image computer-aided diagnostic device, which can be applied to... Figure 1A , Figure 1B and Figure 1C The image processing model shown is as follows: Figure 8As shown, the device includes: a determining module 801, used to determine pruning parameters based on a text feature sequence and an image feature sequence, wherein the image feature sequence is obtained by feature extraction from a plain CT image, and the text feature sequence is obtained by feature extraction from a CT image processing instruction of the plain CT image; a pruning module 802, used to prune the image feature sequence according to the pruning parameters to obtain multiple retained image features; and an acquiring module 803, used to obtain a processing result for responding to the CT image processing instruction based on the multiple retained image features and the text feature sequence.

[0132] The functions of each module in the devices of this application embodiment can be found in the corresponding descriptions of the methods described above, and they have corresponding beneficial effects, which will not be repeated here. Furthermore, the device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components illustrated as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application solution according to actual needs.

[0133] Figure 9 This is a block diagram of an electronic device used to implement embodiments of this application. For example... Figure 9 As shown, the electronic device includes a memory 901 and a processor 902. The memory 901 stores a computer program that can run on the processor 902. When the processor 902 executes the computer program, it implements the method described in the above embodiments. The number of memories 901 and processors 902 can be one or more. In a specific implementation, the electronic device may also include a communication interface 903 for communicating with external devices and exchanging data.

[0134] In practical implementation, if the memory 901, processor 902, and communication interface 903 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0135] Optionally, in a specific implementation, if the memory 901, processor 902, and communication interface 903 are integrated on a single chip, then the memory 901, processor 902, and communication interface 903 can communicate with each other through an internal interface.

[0136] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method provided in this application.

[0137] This application provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in this application.

[0138] This application also provides a chip including a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0139] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0140] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0141] Further, optionally, the aforementioned memory may include read-only memory and random access memory. The memory may be volatile memory or non-volatile memory, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Sync Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0142] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0143] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0144] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0145] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0146] The logic and / or steps described in the flowchart or otherwise herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0147] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0148] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0149] The above description is merely an exemplary embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope described in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method applied to an image processing model, the method comprising: Pruning parameters are determined based on text feature sequences and image feature sequences, wherein the image feature sequences are obtained by extracting features from the image to be processed, and the text feature sequences are obtained by extracting features from the image processing instructions of the image to be processed; The image feature sequence is pruned according to the pruning parameters to obtain multiple retained image features; Based on the multiple preserved image features and the text feature sequence, a processing result is obtained to respond to the image processing command.

2. The method according to claim 1, wherein, The pruning parameters include an index threshold and a saliency index for any image feature in the image feature sequence. The saliency index is used to characterize the degree of correlation between the image feature and the image processing instruction. Pruning the image feature sequence according to the pruning parameters includes: For any image feature in the image feature sequence, if the saliency index of the image feature is less than the index threshold, the image feature is pruned.

3. The method according to claim 2, wherein, The step of determining pruning parameters based on text feature sequences and image feature sequences includes: For a target text feature in the text feature sequence, the weight corresponding to the target text feature is determined based on the correlation between the target text feature and other text features, wherein the target text feature is any text feature in the text feature sequence, and the other text features are all text features in the text feature sequence other than the target text feature; For any image feature in the image feature sequence, the saliency index of the image feature is determined based on the weights corresponding to the image feature and each text feature in the text feature sequence.

4. The method according to claim 3, wherein, The step of determining the weight corresponding to the target text feature based on the correlation between the target text feature and other text features includes: Determine the attention scores between the target text feature and the other text features; The attention scores are aggregated to obtain the aggregated score corresponding to the target text feature; The aggregated scores are subjected to a first transformation to obtain the weights corresponding to the target text features.

5. The method according to claim 3, wherein, The step of determining the saliency index of the image features based on the weights corresponding to each text feature in the image feature and text feature sequence includes: For any text feature in the text feature sequence, determine the similarity between the image feature and the text feature; Based on the weights corresponding to each text feature in the text feature sequence, the corresponding similarities are summed in a weighted manner to obtain the weighted similarity of the image features; A second transformation is performed on the weighted similarity to obtain the saliency index of the image features.

6. The method according to claim 2, wherein, The step of determining pruning parameters based on text feature sequences and image feature sequences includes: The distribution features, scale features, and compression features corresponding to the saliency indices of each image feature in the image feature sequence are determined. The distribution features are used to characterize the overall distribution of each saliency index, the scale features are used to characterize the original range corresponding to each saliency index, and the compression features are used to characterize the concentration of important image features. The index threshold is predicted based on the distribution characteristics, the scale characteristics, and the compression characteristics.

7. A training method for an image processing model, comprising: Acquire multiple training samples, each of which includes a training image and training instructions; The image processing model is obtained by training the network to be trained using the multiple training samples, wherein the image processing model is used to implement the method described in any one of claims 1-6.

8. The method according to claim 7, wherein, The step of training the network to be trained using the multiple training samples to obtain the image processing model includes: The pruning parameters of the training sample are determined based on the text feature sequence and image feature sequence corresponding to any training sample. Generate a surrogate mask for each image feature in the image feature sequence based on the pruning parameters; The image feature sequence is pruned according to the pruning parameters to obtain multiple retained image features; The parameters of the network to be trained are updated based on the multiple preserved image features and the proxy mask to obtain the image processing model.

9. The method according to claim 8, wherein, The pruning parameters include an index threshold and a saliency index for any image feature in the image feature sequence. Generating a surrogate mask for each image feature in the image feature sequence based on the pruning parameters includes: For any image feature in the image feature sequence, the retention probability of the image feature is determined based on the saliency index and index threshold of the image feature; A third transformation is performed on the retained probabilities to obtain the surrogate mask of the image features.

10. The method according to claim 8, wherein, The parameters of the network to be trained include the parameters of the first network layer and the parameters of the second network layer. The first network layer is used for pruning, and the second network layer consists of all network layers in the network to be trained except for the first network layer. Updating the parameters of the network to be trained based on the multiple preserved image features and the proxy mask includes: The surrogate mask is scattered to obtain a gradient tensor, and the gradient tensor has the same dimension as the image feature sequence. Based on the position of the preserved image features in the image feature sequence, the gradient of the preserved image features is filled into the gradient tensor to obtain a new gradient tensor; Update the parameters of the second network layer based on the new gradient tensor; For any image feature in the image feature sequence, a proxy gradient corresponding to the image feature is generated based on the proxy mask of the image feature and the initial feature corresponding to the image feature, wherein the initial feature is the feature input to the network to be trained; The parameters of the first network layer are updated based on the proxy gradients corresponding to each image feature in the image feature sequence.

11. The method according to claim 10, wherein, The step of generating a proxy gradient corresponding to the image feature based on the proxy mask of the image feature and the initial feature corresponding to the image feature includes: Based on the initial features corresponding to the image features, the contribution of the image features is determined, wherein the contribution is used to characterize the impact of deleting the image features on the model performance of the image processing model; Based on the contribution and the surrogate mask of the image feature, a direction factor corresponding to the image feature is determined, wherein the direction factor is used to indicate whether the contribution of the image feature to completing the training instruction is positive or negative; Based on the initial features of the image features and the gradient of the image features in the updated gradient tensor, the magnitude factor corresponding to the image features is determined, wherein the magnitude factor is used to characterize the degree of correlation between the image features and the training instructions; The surrogate gradient of the image feature is determined based on the direction factor and the magnitude factor.

12. A computer-aided diagnostic method for medical images, applied to an image processing model, the method comprising: The pruning parameters are determined based on the text feature sequence and the image feature sequence, wherein the image feature sequence is obtained by extracting features from the plain CT image of the target detection area, and the text feature sequence is obtained by extracting features from the CT image processing instructions of the plain CT image; The image feature sequence is pruned according to the pruning parameters to obtain multiple retained image features; Based on the multiple preserved image features and the text feature sequence, a processing result is obtained to respond to the CT image processing command.

13. A computer-aided cancer diagnosis system, comprising a client and a server; The client is used to send a task processing request to the server, wherein... The task processing request includes a plain CT image of the target detection area and CT image processing instructions for the plain CT image; The server is used to invoke the image processing model to obtain the processing result in response to the CT image processing command, and the image processing model is used to implement the method of claim 12.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor, when executing the computer program, implements the method of any one of claims 1 to 12.

15. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 12.

Images