Image processing method, device, electronic device and storage medium

Abstract

The embodiment of the present invention discloses an image processing method, device, electronic device and storage medium. The method comprises: obtaining the data to be processed of the target lesion; inputting each to-be-processed video frame in the to-be-used classified video, the target static image and the fused image corresponding to each to-be-processed video frame into a pre-trained image fusion model in turn to obtain the target fused image; classifying and processing the target fused image based on the pre-trained image classification model to obtain the classification attribute characteristics corresponding to the to-be-processed data; and determining the target classification result corresponding to the target lesion based on the classification attribute characteristics. The method solves the problem in the prior art that the lesion type corresponding to the image including the lesion is manually identified, resulting in low identification efficiency and poor accuracy while wasting manpower, and achieves the effect of saving manpower while improving the convenience and effectiveness of image classification corresponding to the lesion information.

Description

Technical Field

[0001] The present invention relates to image processing technology, and more particularly to an image processing method, device, electronic device, and storage medium. Background Art

[0002] With the rapid development of computer science and medical imaging engineering, a host of advanced medical imaging devices have emerged, providing medical images for clinical diagnosis. These images can reflect information about the human body's structure, organs, and diseased tissues. Therefore, identifying lesions in images and determining their type has become an increasingly common method. In existing technologies, physicians typically identify lesion types by directly observing the characteristics of lesions in medical images. Alternatively, to achieve intelligent image processing, intelligent algorithms are often used to fuse multiple medical images containing the lesions. The resulting fused image is then handed over to the physician for lesion type determination. The lesion type is often closely related to the physician's clinical experience. However, in cases where lesion information is difficult to identify, for example, in digital radiography (DR) images, even experienced physicians may have difficulty distinguishing whether a thoracic or lumbar compression fracture is old or fresh, resulting in significant waste of manpower and low identification efficiency. Summary of the Invention

[0003] The embodiments of the present invention provide an image processing method, device, electronic device and storage medium to achieve the technical effect of improving the convenience and effectiveness of image classification corresponding to lesion information while saving manpower.

[0004] In a first aspect, an embodiment of the present invention provides an image processing method, the method comprising:

[0005] Acquire data to be processed of a target lesion; wherein the data to be processed includes a target static image corresponding to the target lesion and a classification video to be used, the classification video to be used includes multiple video frames to be processed; the target static image is an image different from the multiple video frames to be processed;

[0006] Inputting each to-be-processed video frame in the to-be-processed classification video, the target static image, and the fused image corresponding to each to-be-processed video frame into a pre-trained image fusion model in sequence to obtain a target fused image; wherein the fused image is obtained by fusing the previous to-be-processed video frame, the fused image corresponding to the previous to-be-processed video frame, and the target static image based on the image fusion model;

[0007] Classify the target fusion image based on a pre-trained image classification model to obtain classification attribute features corresponding to the data to be processed;

[0008] Based on the classification attribute characteristics, a target classification result corresponding to the target lesion is determined.

[0009] In a second aspect, an embodiment of the present invention further provides an image processing device, the device comprising:

[0010] a module for acquiring data to be processed, configured to acquire data to be processed of a target lesion; wherein the data to be processed includes a target static image corresponding to the target lesion and a classification video to be used, wherein the classification video to be used includes a plurality of video frames to be processed; and the target static image is an image different from the plurality of video frames to be processed;

[0011] a target fused image acquisition module, configured to sequentially input each to-be-processed video frame in the to-be-used classification video, the target static image, and the fused image corresponding to each to-be-processed video frame into a pre-trained image fusion model to obtain a target fused image; wherein the fused image is obtained by fusing the previous to-be-processed video frame, the fused image corresponding to the previous to-be-processed video frame, and the target static image based on the image fusion model;

[0012] A classification attribute feature acquisition module is used to classify the target fusion image based on a pre-trained image classification model to obtain classification attribute features corresponding to the data to be processed;

[0013] The target classification result determination module is used to determine the target classification result corresponding to the target lesion based on the classification attribute characteristics.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising:

[0015] one or more processors;

[0016] a storage device for storing one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of the embodiments of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method as described in any one of the embodiments of the present invention.

[0019] The technical solution of the embodiment of the present invention obtains the data to be processed of the target lesion, and inputs each to-be-processed video frame, the target static image and the fused image corresponding to each to-be-processed video frame in the to-be-processed classification video into the image fusion model in turn to obtain the target fused image, and then classifies the target fused image based on the image classification model to obtain the classification attribute features corresponding to the to-be-processed data, and determines the target classification result corresponding to the target lesion based on the classification attribute features. This solves the problem in the prior art of manually identifying the lesion type corresponding to the image including the lesion, which leads to low identification efficiency and poor accuracy while wasting manpower. This solution realizes the fusion of the to-be-processed video frame and the target static image by cyclically processing the to-be-processed video frame and the target static image in the to-be-processed classification video, and obtains a target fused image with high feature richness. Then, the target fused image is intelligently identified based on the image classification model to determine the lesion type corresponding to the target lesion, saving manpower while improving the convenience and effectiveness of image classification corresponding to the lesion information, thereby achieving the technical effect of reducing misdiagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings introduced here only illustrate some of the embodiments to be described by the present invention, and are not exhaustive. A person skilled in the art can derive other drawings based on these drawings without inventive effort.

[0021] Figure 1 This is a flowchart of an image processing method provided in the first embodiment of the present invention;

[0022] Figure 2 A schematic diagram of an image processing method provided in the second embodiment of the present invention;

[0023] Figure 3 A schematic diagram of an image processing method provided in the second embodiment of the present invention;

[0024] Figure 4 A schematic diagram of an image processing method provided in the second embodiment of the present invention;

[0025] Figure 5 A schematic diagram of an image processing method provided in Embodiment 3 of the present invention;

[0026] Figure 6 A schematic diagram of an image processing method provided in Embodiment 3 of the present invention;

[0027] Figure 7 A schematic diagram of an image processing method provided in Embodiment 3 of the present invention;

[0028] Figure 8 This is a structural block diagram of an image processing device provided by a fourth embodiment of the present invention;

[0029] Figure 9 This is a structural diagram of an electronic device provided in Example 5 of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0031] Example 1

[0032] Figure 1 This is a flowchart of an image processing method provided in Embodiment 1 of the present invention. This embodiment is applicable to image classification. The method can be performed by an image processing device in an embodiment of the present invention. The device can be implemented using software and / or hardware. Optionally, it can be implemented by an electronic device, such as a mobile terminal, a PC, or a server. The device can be configured in a computing device. The image processing method provided in this embodiment specifically includes the following steps:

[0033] S110: Obtain data to be processed of the target lesion.

[0034] The data to be processed includes a target static image corresponding to the target lesion and a classification video to be used. The classification video to be used includes multiple video frames to be processed; the target static image is an image distinct from the multiple video frames to be processed. A target lesion refers to any location where information on a syndrome or infection is desired to be collected, such as a vascular lesion or a keratitis lesion. For example, a portion of the cornea that is infected by inflammation is considered a keratitis lesion. The target static image can be understood as an image to be processed. This image can be captured by the application software or pre-stored by the application software from a storage space. In actual applications, an image including the target lesion can be captured by the application software. In this case, the lesion type corresponding to the target lesion can be determined based on image processing. The classification video to be used can be image data generated by large medical imaging equipment including CT, MRI, PET-CT, slit lamp, fluorescein angiography, etc., or small imaging equipment. It can be data in DICOM format or AVI format. In practical applications, a slit lamp can be used to photograph the fundus of a patient to obtain a video image including fundus lesions as a classification video to be used. At this time, the lesion type of the patient's fundus lesion can be obtained based on the video processing.

[0035] Specifically, a camera device can be used to photograph the target lesion, obtaining a target static image including the target lesion as the data to be processed. Alternatively, an imaging device can be used to photograph the target lesion, obtaining a classification video including the target lesion as the data to be processed. The system can receive multiple channels of data to be processed, each corresponding to a specific target lesion. These data can include sequence data in the form of a video, i.e., the classification video to be used, or static data in the form of an image, i.e., the target static image. This allows the system to process each channel of data to be processed according to its internal model algorithm and obtain the lesion type corresponding to the target lesion.

[0036] S120 , sequentially inputting each to-be-processed video frame in the to-be-used classification video, the target static image, and the fused image corresponding to each to-be-processed video frame into a pre-trained image fusion model to obtain a target fused image.

[0037] The fused image is obtained by fusing the previous video frame to be processed, the fused image corresponding to the previous video frame to be processed, and the target static image using an image fusion model. The image fusion model can be understood as a pre-trained model used to fuse the various input data to be processed.

[0038] In practical applications, each of the to-be-processed video frames in the to-be-classified video can be sequentially input into the image fusion model, such as the first frame, the second frame, the third frame, and the fourth frame. At the same time, the target static image can also be input into the image fusion model. At this time, the first frame of the to-be-processed video frame and the target static image can be fused to obtain a fused image corresponding to the first frame. Furthermore, the second frame of the to-be-processed video frame, the target static image, and the fused image corresponding to the first frame can be fused to obtain a fused image corresponding to the second frame. Accordingly, the to-be-processed video frame corresponding to the last frame, the target static image, and the fused image corresponding to the previous frame can be fused to obtain a fused image. This fused image can be used as the target fused image to determine the lesion type corresponding to the target lesion based on the target fused image, thereby improving the accuracy of image classification.

[0039] It should be noted that in order to improve the accuracy of image fusion in space, before inputting each to-be-processed video frame, the target static image and the fused image corresponding to each to-be-processed video frame in the to-be-processed classification video into a pre-trained image fusion model in turn and obtaining the target fused image, each to-be-processed video frame and the target static image in the input classification video to be used can be processed, such as downsampling processing. The downsampling technology can be used to reduce the data size corresponding to the to-be-processed video frame and the target static image, thereby improving the spatial concentration of the data. The video frame to be processed and the target static image after centralized processing can be input into the image fusion model to obtain the target fused image.

[0040] Optionally, before sequentially inputting each to-be-processed video frame in the to-be-used classified video, the target static image, and the fused image corresponding to each to-be-processed video frame into a pre-trained image fusion model to obtain the target fused image, the method includes: inputting each to-be-processed video frame and the target static image in the to-be-used classified video into a corresponding image preprocessing sub-model to obtain each processed to-be-processed video frame and the target static image;

[0041] The image preprocessing submodel is used to reduce the size of the video frame to be processed and the target static image, as well as to increase the number of feature layers for the video frame to be processed and the target static image. In practical applications, corresponding image preprocessing submodels can be preconfigured for the input video frame to be processed and the target static image. For example, a target static image captured by a camera device A can correspond to an image preprocessing submodel A, which is used to reduce the size of the target static image and increase the number of feature layers for the target static image. A video to be classified, captured by a slit lamp B, can correspond to an image preprocessing submodel B. A video to be classified, captured by a fluorescein angiography device C, can correspond to an image preprocessing submodel C. It should be noted that each video to be classified and / or target static image can correspond to an image preprocessing submodel, and this can also be determined by technicians based on actual work conditions. This is not limited in this technical solution. Optionally, each image preprocessing submodel can be a 4-8 layer CNN (Convolutional Neural Network), with a CNN architecture consisting of one unit every 2-4 layers, with downsampling layers between units. For example, each video frame to be processed and the target static image in the classification video to be used can be input into the corresponding image preprocessing sub-model, the image size can be reduced to 1 / 4 of the original, and the number of feature layers can be increased to more than 64 layers.

[0042] Specifically, each unprocessed video frame in the video to be classified can be used as input to the corresponding image preprocessing sub-model for the video to be classified. This can output each unprocessed video frame with its image size reduced to 1 / 4 of its original size and the number of feature layers increased to 64 or more. A target static image can also be used as input to the corresponding image preprocessing sub-model, resulting in a target static image with a reduced image size and increased feature layers. By inputting the processed video frames to be processed and the target static image into the image fusion model, the accuracy of image fusion can be improved.

[0043] S130 , classifying the target fusion image based on a pre-trained image classification model to obtain classification attribute features corresponding to the data to be processed.

[0044] The image classification model can be understood as a preset model for classifying images, and the classification attribute feature can be understood as the feature value corresponding to the classification result.

[0045] In this embodiment, the target fusion image corresponding to the target lesion can be input into a pre-trained image classification model, and the classification results and corresponding feature values ​​of the preset number of classifications can be output according to the preset lesion type requirements. For example, assuming that the preset classification results are two items: normal symptoms and early stage symptoms, the image classification model at this time is a two-class classification model. For example, a probability value of 80% corresponding to normal symptoms can be output, and 80% can be used as a classification attribute feature. A probability value of 20% corresponding to the early stage symptoms can also be output, and 20% can also be used as a classification attribute feature. Accordingly, the classification attribute features corresponding to each preset classification result can be obtained, so that the final classification result can be determined based on each classification attribute feature.

[0046] S140: Determine a target classification result corresponding to the target lesion based on the classification attribute characteristics.

[0047] The preset classification result may be at least one of normal condition, early stage condition, mid-stage condition, and late stage condition. Optionally, the target classification result may include normal condition, early stage condition, mid-stage condition, or late stage condition. It should be noted that the preset classification result can be determined by technical personnel based on actual work conditions and is not limited in this technical solution.

[0048] Specifically, after obtaining each preset classification result and the corresponding classification attribute feature, each classification attribute feature can be compared. For example, assuming that the classification attribute feature 1 corresponding to the normal condition is 80% and the classification attribute feature 2 corresponding to the early stage of the condition is 20%, then the normal condition corresponding to the maximum value of 80% can be used as the target classification result corresponding to the target lesion.

[0049] The technical solution of this embodiment obtains the data to be processed of the target lesion, and inputs each to-be-processed video frame in the to-be-processed classification video, the target static image and the fused image corresponding to each to-be-processed video frame into the image fusion model in turn to obtain the target fused image, and then classifies the target fused image based on the image classification model to obtain the classification attribute features corresponding to the to-be-processed data, and determines the target classification result corresponding to the target lesion based on the classification attribute features. This solves the problem in the prior art of manually identifying the lesion type corresponding to the image including the lesion, which leads to low identification efficiency and poor accuracy while wasting manpower. This solution realizes the fusion of the to-be-processed video frame and the target static image by cyclically processing the to-be-processed video frame in the to-be-processed classification video, and obtains a target fused image with high feature richness. Then, the target fused image is intelligently identified based on the image classification model to determine the lesion type corresponding to the target lesion, saving manpower while improving the convenience and effectiveness of image classification corresponding to lesion information, thereby achieving the technical effect of reducing misdiagnosis.

[0050] Example 2

[0051] Figure 2 This is a schematic diagram of an image processing method provided in Example 2 of the present invention. Based on the previous example, S120 is further refined. For specific implementations, please refer to the technical solution of this example. Technical terms that are identical or corresponding to those in the previous example are not repeated here.

[0052] like Figure 2 As shown, the method specifically includes the following steps:

[0053] S210: Obtain data to be processed of the target lesion.

[0054] S220: Input the first to-be-processed video frame in the to-be-used classified video and the target static image into an image fusion model to obtain a first fused image corresponding to the first to-be-processed video frame as a current fused image.

[0055] In this embodiment, when each to-be-processed video frame in the to-be-processed classification video, the target static image and the fused image corresponding to each to-be-processed video frame are sequentially input into a pre-trained image fusion model, the first to-be-processed video frame in the to-be-processed classification video can first be input into the image fusion model as the first to-be-processed video frame and the target static image, and the first to-be-processed video frame and the target static image can be fused to obtain a fused image corresponding to the first to-be-processed video frame, that is, a first fused image. At this time, the first fused image can be used as the current fused image to further obtain the current fused image, the second to-be-processed video frame in the to-be-processed classification video and the fused image corresponding to the target static image.

[0056] It should be noted that, based on the above technical solution, in order to clearly introduce how the image fusion model determines the first fused image of the first video frame to be processed and the target static image, see Figure 3 ,in Figure 3 Box 1 in the figure can represent an image fusion model. Optionally, the image fusion model can include a weight assignment submodel, a target fusion submodel, and a feature processing submodel. The weight assignment submodel is used to determine the fusion weights for each channel of processed data. The target fusion submodel is used to fuse the various channels of processed data and output a fused image. The feature processing submodel is used to reduce the size of the fused image and increase the number of feature layers in the fused image. Optionally, the feature processing submodel can be an 8-16-layer CNN, with a unit consisting of four layers in the CNN architecture, and downsampling layers between units. In practical applications, the first processed video frame and the target static image in the video to be classified can be used as inputs to the weight assignment submodel, which outputs the fusion weights corresponding to the first processed video frame and the target static image. The first processed video frame, the target static image, and the corresponding fusion weights can be input to the target fusion submodel. The target fusion submodel can use various fusion methods, such as weighted summation of the image and the corresponding fusion weights or weighted averaging of the image and the corresponding fusion weights. Correspondingly, a fused image corresponding to the first video frame to be processed can be obtained. At this time, the fused image can be used as the input of the feature processing sub-model, and the fused image after reducing the size and increasing the number of feature layers is output as the first fused image.

[0057] Based on the above technical solution, in order to clearly introduce how the weight allocation sub-model determines the fusion weight value of the first video frame to be processed and the target static image, see Figure 4 , Figure 4 Box 2 in the figure can be represented as a weight allocation sub-model. Optionally, the weight allocation sub-model can include a feature preprocessing module, a feature mapping module, a normalization module and a weight allocation module. Figure 4 Box 3 in the figure can be represented as a feature preprocessing module, which includes a feature compression submodule and a spatial partitioning submodule. The feature compression submodule is used to compress the data to be processed to obtain the full-image feature data corresponding to the data to be processed. For example, if the data to be processed is in a 64×64×64 format, the full-image feature data after feature compression processing can be one-dimensional vector feature data. The spatial partitioning submodule is used to partition the data to be processed to obtain the spatial feature data corresponding to the data to be processed. For example, if the data to be processed is in a 64×64×64 format, if it is divided into several small blocks according to space, such as 4×4×4 blocks, each small block is 16×16×16. It should be noted that in order to improve the accuracy of image fusion, the number of small blocks in which the image is divided should be greater than 8×8×8. In practical applications, the first video frame to be processed and the target static image can be used as inputs to a feature preprocessing module, which outputs global and spatial feature data corresponding to the first video frame to be processed and the target static image. The global feature data corresponding to the target static image and the spatial feature data corresponding to the first video frame to be processed can then be used as inputs to a feature mapping module. Optionally, the feature mapping module can employ a multi-layer fully connected neural network. A projection layer calculates the attention data of the target static image with respect to the first video frame to be processed. For example, attention data is obtained by combining each small block of data with the global feature data. In this case, the attention data is also spatially divided into multiple small blocks. Accordingly, attention data of the first video frame to be processed with respect to the target static image can also be obtained. The two attention data can be used as inputs to a normalization module. Optionally, the normalization module can employ a softmax (logistic regression model). Softmax can be used to implement normalization. For example, based on each attention data block, if the image ratios are not uniform across different blocks, they can be automatically adjusted based on the features of different locations in the image. Accordingly, the sum of blocks with the same coordinate position in each attention data block is 1. Normalized attention data can thus be obtained. The normalized attention data can be used as a weight distribution module, and the fusion weight value corresponding to the first video frame to be processed and the fusion weight value corresponding to the target static image can be output, so that the first video frame to be processed, the target static image and the corresponding fusion weight value can be input into the target fusion sub-model to obtain the first fused image corresponding to the first video frame to be processed, and used as the current fused image.

[0058] S230. Use the current fused image, the second video frame to be processed in the classification video to be used, and the target static image as inputs of the image fusion model to obtain a second fused image corresponding to the second video frame to be processed, and update the second fused image to the current fused image.

[0059] Specifically, the current fused image, the second video frame to be processed in the classification video to be used, and the target static image can be input into the image fusion model, and the current fused image, the second video frame to be processed, and the target static image can be fused to obtain a fused image corresponding to the second video frame to be processed, that is, a second fused image. At this time, the second fused image can be used as the new current fused image.

[0060] It should be noted that the current fused image, the second video frame to be processed in the classification video to be used, and the target static image are used as inputs of the image fusion model to obtain a second fused image corresponding to the second video frame to be processed, and the second fused image is updated to the current fused image. The fusion weight values ​​corresponding to the current fused image, the second video frame to be processed in the classification video to be used, and the target static image can also be determined based on the weight allocation sub-model to determine the second fused image based on the fusion weight values.

[0061] Optionally, the current fused image, the second video frame to be processed in the classification video to be used, and the target static image are used as inputs of the image fusion model to obtain a second fused image corresponding to the second video frame to be processed, and the second fused image is updated to the current fused image, including: inputting the current fused image, the second video frame to be processed in the classification video to be used, and the target static image into the weight allocation sub-model to obtain fusion weight values ​​corresponding to the current fused image, the second video frame to be processed, and the target static image; inputting the current fused image, the second video frame to be processed, the target static image, and the corresponding fusion weight values ​​into the target fusion sub-model to obtain the current fused image to be used corresponding to the second video frame to be processed; inputting the current fused image to be used into the feature processing sub-model to obtain a second fused image corresponding to the second video frame to be processed, and updating the second fused image to the current fused image.

[0062] In this embodiment, after the first fused image corresponding to the first video frame to be processed is used as the current fused image, the current fused image at this time, the second video frame to be processed in the classification video to be used, and the target static image can be used as inputs of the weight allocation sub-model, and the fusion weight values ​​corresponding to the current fused image, the second video frame to be processed, and the target static image can be output. The current fused image, the second video frame to be processed, the target static image, and the corresponding fusion weight values ​​can be used as inputs of the target fusion sub-model to obtain a fused image corresponding to the second video frame to be processed, that is, the current fused image to be used. At this time, the current fused image to be used can be used as input of the feature processing sub-model to obtain a fused image corresponding to the second video frame to be processed, that is, the second fused image. The second fused image can be used as the new current fused image, so that the fused image corresponding to the third video frame to be processed can be determined based on the new current fused image.

[0063] On the basis of the above technical solution, in order to clearly introduce how the weight allocation sub-model determines the fusion weight values ​​of the current fused image, the second video frame to be processed and the target static image. Optionally, the current fused image, the second video frame to be processed in the classification video to be used, and the target static image are input into the weight allocation sub-model to obtain the fusion weight values ​​corresponding to the current fused image, the second video frame to be processed, and the target static image, including: inputting the current fused image, the second video frame to be processed in the classification video to be used, and the target static image into the feature preprocessing module to obtain the full-image feature data and spatial feature data corresponding to the current fused image, the second video frame to be processed, and the target static image; inputting the full-image feature data and spatial feature data corresponding to the current fused image, the second video frame to be processed, and the target static image into the feature mapping module to obtain the attention data to be processed corresponding to the current fused image, the second video frame to be processed, and the target static image; inputting the attention data to be processed corresponding to the current fused image, the second video frame to be processed, and the target static image into the normalization module to obtain the target attention data corresponding to the current fused image, the second video frame to be processed, and the target static image; processing the target attention data corresponding to the current fused image, the second video frame to be processed, and the target static image based on the weight allocation module to obtain the fusion weight values ​​corresponding to the current fused image, the second video frame to be processed, and the target static image.

[0064] In this embodiment, the current fused image, the second video frame to be processed in the classification video to be used, and the target static image can be used as inputs of the feature preprocessing module, and the full-image feature data and spatial feature data corresponding to the current fused image, the second video frame to be processed, and the target static image can be output. The full-image feature data and spatial feature data corresponding to the current fused image, the second video frame to be processed, and the target static image can be input into the feature mapping module to obtain the attention data corresponding to the current fused image, the second video frame to be processed, and the target static image, that is, the attention data to be processed. The attention data to be processed corresponding to the current fused image, the second video frame to be processed, and the target static image can be used as inputs of the normalization module. For example, the small blocks corresponding to the coordinate positions of the six attention data can be processed so that the sum is 1, and the six normalized attention data can be obtained. In other words, the normalized attention data corresponding to the current fused image, the second video frame to be processed, and the target static image, that is, the target attention data, can be obtained. The target attention data can be input into the weight distribution module to obtain the fusion weight values ​​corresponding to the current fused image, the second video frame to be processed and the target static image, so that the second fused image corresponding to the second video frame to be processed can be determined based on the fusion weight values.

[0065] It should be noted that the current fused image to be used is input into the feature processing sub-model to obtain a second fused image corresponding to the second video frame to be processed, and the second fused image is updated to the current fused image. In order to ensure that when the current fused image is fused with the third video frame to be processed and the target static image, the three images are all images of uniform specifications, optionally, when the current fused image is obtained, the image can be processed, such as feature processing, and the algorithm can be used to process the current fused image into a format consistent with the video frame to be processed and the target static image, thereby improving the efficiency of fusion.

[0066] Optionally, the current fused image to be used is input into the feature processing sub-model to obtain a second fused image corresponding to the second video frame to be processed, and the second fused image is updated to the current fused image. It also includes: inputting the current fused image to be used into the feature processing sub-model to obtain a second fused image to be processed corresponding to the second video frame to be processed; if the second video frame to be processed is not the last video frame to be processed in the classified video to be used, then upsampling the second fused image to be processed to obtain a second fused image, and updating the second fused image to the current fused image.

[0067] In this embodiment, the current fused image to be used can be input into the feature processing sub-model to obtain a second fused image to be processed corresponding to the second video frame to be processed. At this time, if the second video frame to be processed is the last video frame to be processed in the classification video to be used, the second fused image can be used as the final target fused image. If the second video frame to be processed is not the last video frame to be processed in the classification video to be used, the second fused image to be processed can be upsampled using an upsampling algorithm, such as to make the image size of the second fused image to be processed the same as the image size of the third video frame to be processed. The upsampled image can be used as the second fused image, and the second fused image at this time can be used as the current fused image until the last video frame to be processed in the classification video to be used is calculated, and the target fused image is obtained.

[0068] S240 , sequentially executing the image fusion model on each to-be-processed video frame, using the current fused image corresponding to the previous video frame, the current to-be-processed video frame, and the target static image as inputs to obtain a target fused image.

[0069] In this embodiment, the current fused image corresponding to the previous video frame, the current video frame to be processed, and the target static image can be sequentially processed for each video frame to be processed, so as to process the input images cyclically based on the image fusion model to obtain the target fused image. For example, in an actual application, a video to be classified includes frames at time t1, t2, t3, ..., t n The video frame to be processed, where n is a normal natural number, the video frame corresponding to t1 is the first video frame to be processed, the video frame corresponding to t2 is the second video frame to be processed, and t n The corresponding video frame is the last video frame to be processed. First, the first video frame to be processed and the target static image can be used as inputs to the image fusion model to obtain the corresponding first fused image. Then, the first fused image can be used as the current fused image and the second video frame to be processed and the target static image can be used as inputs to the image fusion model to obtain the corresponding second fused image. The fused image obtained at the end of each step is cyclically input into the image fusion model, and the next video frame to be processed is synchronously input. Accordingly, the final fused image corresponding to the last video frame to be processed can be obtained as the target fused image.

[0070] In practical applications, before inputting the classification video to be used and the target static image into the image fusion model, type identifiers corresponding to the different data types of the data to be processed can be preconfigured and stored in a library of data type indicators. For example, the type identifier of the classification video to be used can be dynamic, while the type identifier of the target static image can be static. This allows the model to search for the corresponding identifier from the pre-stored morphological feature library based on the user's selection on the interface and input it along with the data to be processed, providing guidance for the model to process data and improving data processing efficiency.

[0071] S250 , classifying the target fusion image based on a pre-trained image classification model to obtain classification attribute features corresponding to the data to be processed.

[0072] S260: Determine a target classification result corresponding to the target lesion based on the classification attribute characteristics.

[0073] The technical solution of this embodiment obtains the data to be processed of the target lesion, and inputs each to-be-processed video frame in the to-be-processed classification video, the target static image and the fused image corresponding to each to-be-processed video frame into the image fusion model in turn to obtain the target fused image, and then classifies the target fused image based on the image classification model to obtain the classification attribute features corresponding to the to-be-processed data, and determines the target classification result corresponding to the target lesion based on the classification attribute features. This solves the problem in the prior art of manually identifying the lesion type corresponding to the image including the lesion, which leads to low identification efficiency and poor accuracy while wasting manpower. This solution realizes the fusion of the to-be-processed video frame and the target static image by cyclically processing the to-be-processed video frame in the to-be-processed classification video, and obtains a target fused image with high feature richness. Then, the target fused image is intelligently identified based on the image classification model to determine the lesion type corresponding to the target lesion, saving manpower while improving the convenience and effectiveness of image classification corresponding to lesion information, thereby achieving the technical effect of reducing misdiagnosis.

[0074] Example 3

[0075] As an alternative embodiment of the above embodiment, Figure 5 This is a schematic diagram of an image processing method provided by Embodiment 3 of the present invention. For details, please refer to the following specific content.

[0076] See also Figure 5, the first video frame to be processed and the target static image in the video to be classified can be input into the corresponding image preprocessing sub-model. For example, each image preprocessing sub-model is a 4-8 layer CNN network with a structure of one unit every 2-4 layers, and a downsampling layer is set between the units. After the first video frame to be processed and the target static image pass through their respective specific image preprocessing sub-models, the image size can be reduced to 1 / 4 of the original size, and the number of feature layers can reach more than 64. Furthermore, the output of the image preprocessing sub-model can be used as the input of the feature preprocessing module, wherein the feature preprocessing module includes a feature compression sub-module and a spatialized molecular model, which can output the global feature data and spatial feature data corresponding to the first video frame to be processed and the target static image. For example, the image feature data of the first video frame to be processed is divided into several small blocks according to space, and the spatial feature data corresponding to the first video frame to be processed is obtained. Furthermore, the global feature data and spatial feature data corresponding to the first video frame to be processed and the target static image can be input into the feature mapping module. Optionally, the feature mapping module can use a multi-layer fully connected neural network. For example, through a projection layer, each small block in the spatial feature data is combined with the global features of the image to obtain an attention value as the attention data. At this time, the attention data of the first video frame to be processed on the target static image is also divided into multiple small blocks. The same method is used to implement the module's attention data of the target static image on the first video frame to be processed. Accordingly, the attention data corresponding to the first video frame to be processed and the target static image can be obtained. Furthermore, the attention data corresponding to the first video frame to be processed and the target static image can be input into a normalization module. Optionally, the normalization module can be a softmax, and the two attention data can be normalized by softmax. For example, based on each image block, that is, the ratio between the images in different blocks is not the same, and it can be automatically adjusted according to the features of different parts of the image. The number of small blocks into which the image is divided should be 8x8 or more. Accordingly, the normalized attention data can be obtained, and the processed attention data can be input into a weight distribution module to obtain the fusion weight value corresponding to the first video frame to be processed and the target static image, wherein the sum of the fusion weight values ​​of the two is 1. The fusion weights serve as a guide for fusion. The first video frame to be processed, the target static image, and the corresponding fusion weights are input into the target fusion sub-model to generate a single piece of data. For example, the target fusion sub-model can adopt various forms, such as weighted summation using fusion ratios. This data is then input into the feature processing sub-model, which outputs the first fused image. The feature processing sub-model can be a CNN with 8 to 16 layers, with one unit per 4 layers and downsampling layers between units.

[0077] Based on the above technical solution, see Figure 6After obtaining the first fused image, the first fused image can be input as the current fused image and the second video frame to be processed and the target static image into the weight allocation submodule, and the fusion weight values ​​corresponding to the current fused image, the second video frame to be processed and the target static image are output (the sum of the fusion weight values ​​of the three is 1). Furthermore, the current fused image, the second video frame to be processed and the target static image and the corresponding fusion weight values ​​can be input into the target fusion submodel, and the output of the target fusion submodel is used as the input of the feature processing submodel to obtain the second fused image corresponding to the second video frame to be processed. At this time, if the second fused image is not the fused image corresponding to the last video frame to be processed, the fused image can be upsampled so that the fused image is consistent with the image format of the next video frame to be processed. The upsampled fused image can be used as the current fused image. Return to the weight allocation submodule, and process it together with the third video frame to be processed and the target static image through each model until the fused image corresponding to the last video frame to be processed output by the feature processing submodel is completed, and the fused image can be used as the target fused image. The target fusion image can be used as the input of the image classification model. Optionally, the image classification model can be a three-layer fully connected layer, and finally the classification result corresponding to the target fusion image is obtained.

[0078] Based on the above technical solution, see Figure 7 . The type identifiers corresponding to the data to be processed of different data types can be pre-configured and stored in the type indicator library of the data to be processed. For example, the type identifier of the classification video to be used is dynamic, and the type identifier of the target static image is static. When the classification video to be used and the target static image are input into the image fusion model, the corresponding identifier is searched from the pre-stored morphological feature library according to the user's selection on the interface, and is input together with the data to be processed to provide guidance for the model to process the data and improve the efficiency of data processing.

[0079] The technical solution of this embodiment obtains the data to be processed of the target lesion, and inputs each to-be-processed video frame in the to-be-processed classification video, the target static image and the fused image corresponding to each to-be-processed video frame into the image fusion model in turn to obtain the target fused image, and then classifies the target fused image based on the image classification model to obtain the classification attribute features corresponding to the to-be-processed data, and determines the target classification result corresponding to the target lesion based on the classification attribute features. This solves the problem in the prior art of manually identifying the lesion type corresponding to the image including the lesion, which leads to low identification efficiency and poor accuracy while wasting manpower. This solution realizes the fusion of the to-be-processed video frame and the target static image by cyclically processing the to-be-processed video frame in the to-be-processed classification video, and obtains a target fused image with high feature richness. Then, the target fused image is intelligently identified based on the image classification model to determine the lesion type corresponding to the target lesion, saving manpower while improving the convenience and effectiveness of image classification corresponding to lesion information, thereby achieving the technical effect of reducing misdiagnosis.

[0080] Example 4

[0081] Figure 8 This is a structural block diagram of an image processing device provided by the fourth embodiment of the present invention. The device comprises: a to-be-processed data acquisition module 310 , a target fusion image acquisition module 320 , a classification attribute feature acquisition module 330 , and a target classification result determination module 340 .

[0082] Among them, the data to be processed acquisition module 310 is used to acquire the data to be processed of the target lesion; wherein, the data to be processed includes a target static image corresponding to the target lesion and a classification video to be used, and the classification video to be used includes multiple video frames to be processed; the target static image is an image different from the multiple video frames to be processed; the target fusion image acquisition module 320 is used to input each video frame to be processed in the classification video to be used, the target static image and the fusion image corresponding to each video frame to be processed into a pre-trained image fusion model in sequence to obtain a target fusion image; wherein, the fusion image is obtained after the image fusion model fuses the previous video frame to be processed, the fusion image corresponding to the previous video frame to be processed and the target static image; the classification attribute feature acquisition module 330 is used to classify the target fusion image based on a pre-trained image classification model to obtain classification attribute features corresponding to the data to be processed; the target classification result determination module 340 is used to determine the target classification result corresponding to the target lesion based on the classification attribute features.

[0083] The technical solution of this embodiment obtains the data to be processed of the target lesion, and inputs each to-be-processed video frame in the to-be-processed classification video, the target static image and the fused image corresponding to each to-be-processed video frame into the image fusion model in turn to obtain the target fused image, and then classifies the target fused image based on the image classification model to obtain the classification attribute features corresponding to the to-be-processed data, and determines the target classification result corresponding to the target lesion based on the classification attribute features. This solves the problem in the prior art of manually identifying the lesion type corresponding to the image including the lesion, which leads to low identification efficiency and poor accuracy while wasting manpower. This solution realizes the fusion of the to-be-processed video frame and the target static image by cyclically processing the to-be-processed video frame in the to-be-processed classification video, and obtains a target fused image with high feature richness. Then, the target fused image is intelligently identified based on the image classification model to determine the lesion type corresponding to the target lesion, saving manpower while improving the convenience and effectiveness of image classification corresponding to lesion information, thereby achieving the technical effect of reducing misdiagnosis.

[0084] Based on the above device, optionally, the device further includes: an image preprocessing module.

[0085] An image preprocessing module is used to input each to-be-processed video frame and target static image in the to-be-processed classification video into a corresponding image preprocessing sub-model to obtain each to-be-processed video frame and target static image after processing; wherein, the image preprocessing sub-model is used to reduce the size of the to-be-processed video frame and target static image and increase the number of feature layers of the to-be-processed video frame and target static image.

[0086] On the basis of the above device, optionally, the target fused image acquisition module 320 includes a current fused image determination unit, a current fused image update unit and a target fused image determination unit.

[0087] a current fused image determining unit, configured to input a first to-be-processed video frame in the to-be-used classification video and a target static image into an image fusion model, and obtain a first fused image corresponding to the first to-be-processed video frame as the current fused image;

[0088] a current fused image updating unit, configured to use the current fused image, the second to-be-processed video frame in the to-be-used classification video, and the target static image as inputs to the image fusion model, obtain a second fused image corresponding to the second to-be-processed video frame, and update the second fused image as the current fused image;

[0089] The target fused image determination unit is used to sequentially execute the current fused image corresponding to the previous video frame, the current video frame to be processed and the target static image as inputs of the image fusion model to obtain the target fused image.

[0090] Based on the above-mentioned device, optionally, the image fusion model includes a weight allocation sub-model, a target fusion sub-model and a feature processing sub-model, and the current fusion image update unit includes a fusion weight value determination sub-unit, a current fusion image to be used determination sub-unit and a current fusion image update sub-unit.

[0091] a fusion weight value determination subunit, configured to input the current fused image, the second to-be-processed video frame in the to-be-used classification video, and the target static image into the weight allocation submodel, and obtain fusion weight values ​​corresponding to the current fused image, the second to-be-processed video frame, and the target static image;

[0092] a current fused image to be used determining subunit, configured to input the current fused image, the second video frame to be processed, the target static image, and the corresponding fusion weight values ​​into the target fusion submodel to obtain the current fused image to be used corresponding to the second video frame to be processed;

[0093] The current fused image updating subunit is used to input the current fused image to be used into the feature processing submodel to obtain a second fused image corresponding to the second video frame to be processed, and update the second fused image to the current fused image.

[0094] Based on the above-mentioned device, optionally, the weight allocation sub-model includes a feature preprocessing module, a feature mapping module, a normalization module and a weight allocation module, and the fusion weight value determination sub-unit includes a feature preprocessing sub-unit, a to-be-processed attention data determination sub-unit, a target attention data determination sub-unit and a fusion weight value determination sub-unit.

[0095] A feature preprocessing subunit is used to input the current fused image, the second video frame to be processed in the classification video to be used, and the target static image into the feature preprocessing module to obtain full-image feature data and spatial feature data corresponding to the current fused image, the second video frame to be processed, and the target static image;

[0096] a small unit for determining attention data to be processed, configured to input the full-image feature data and spatial feature data corresponding to the current fused image, the second video frame to be processed, and the target static image into the feature mapping module, and obtain the attention data to be processed corresponding to the current fused image, the second video frame to be processed, and the target static image;

[0097] a target attention data determination subunit, configured to input the attention data to be processed corresponding to the current fused image, the second video frame to be processed, and the target static image into the normalization module, and obtain the target attention data corresponding to the current fused image, the second video frame to be processed, and the target static image;

[0098] The fusion weight value determination unit is used to process the target attention data corresponding to the current fused image, the second video frame to be processed and the target static image based on the weight distribution module to obtain the fusion weight values ​​corresponding to the current fused image, the second video frame to be processed and the target static image.

[0099] On the basis of the above device, optionally, the current fused image updating subunit includes a second to-be-processed fused image determining subunit and a current fused image updating subunit.

[0100] A second fused image to be processed determining subunit is configured to input the current fused image to be used into the feature processing submodel to obtain a second fused image to be processed corresponding to the second video frame to be processed;

[0101] The current fused image updating subunit is used to perform upsampling processing on the second to-be-processed fused image to obtain a second fused image if the second to-be-processed video frame is not the last to-be-processed video frame in the classification video to be used, and update the second fused image to the current fused image.

[0102] On the basis of the above device, optionally, the target classification result determination module 340 includes a target classification result determination unit.

[0103] The target classification result determination unit is used to determine the target classification result corresponding to the target lesion based on the classification attribute characteristics; wherein the target classification result includes normal condition, early condition, middle condition or late condition.

[0104] The image processing device provided by the embodiment of the present invention can execute the image processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0105] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of the present invention.

[0106] Example 5

[0107] Figure 9This is a structural diagram of an electronic device provided in Example 5 of the present invention. Figure 9 A block diagram of an exemplary electronic device 40 suitable for implementing exemplary embodiments of the present invention is shown. Figure 9 The electronic device 40 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0108] like Figure 9 As shown, electronic device 40 is a general-purpose computing device. Components of electronic device 40 may include, but are not limited to, one or more processors or processing units 401, system memory 402, and a bus 403 connecting various system components (including system memory 402 and processing unit 401).

[0109] Bus 403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0110] The electronic device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 40, including volatile and non-volatile media, removable and non-removable media.

[0111] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. Electronic device 40 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 406 may be used to read and write non-removable, non-volatile magnetic media ( Figure 9 Not shown, often called a "hard drive"). Although Figure 9 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 403 via one or more data medium interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0112] A program / utility 408 having a set (at least one) of program modules 407 may be stored, for example, in memory 402. Such program modules 407 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 407 generally perform the functions and / or methods of the embodiments described herein.

[0113] The electronic device 40 may also communicate with one or more external devices 409 (e.g., keyboard, pointing device, display 410, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 40, and / or any device that enables the electronic device 40 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 411. Furthermore, the electronic device 40 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 412. As shown, the network adapter 412 communicates with other modules of the electronic device 40 via the bus 403. It should be understood that although Figure 9 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 40, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0114] The processing unit 401 executes various functional applications and data processing by running programs stored in the system memory 402 , such as implementing the image processing method provided by the embodiment of the present invention.

[0115] Example 6

[0116] Embodiment 6 of the present invention further provides a storage medium containing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform an image processing method. The method includes:

[0117] Acquire data to be processed of a target lesion; wherein the data to be processed includes a target static image corresponding to the target lesion and a classification video to be used, the classification video to be used includes multiple video frames to be processed; the target static image is an image different from the multiple video frames to be processed;

[0118] Inputting each to-be-processed video frame in the to-be-processed classification video, the target static image, and the fused image corresponding to each to-be-processed video frame into a pre-trained image fusion model in sequence to obtain a target fused image; wherein the fused image is obtained by fusing the previous to-be-processed video frame, the fused image corresponding to the previous to-be-processed video frame, and the target static image based on the image fusion model;

[0119] Classify the target fusion image based on a pre-trained image classification model to obtain classification attribute features corresponding to the data to be processed;

[0120] Based on the classification attribute characteristics, a target classification result corresponding to the target lesion is determined.

[0121] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0122] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0123] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0124] The computer program code for performing the operations of the embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0125] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire data to be processed of a target lesion; wherein the data to be processed includes a target static image corresponding to the target lesion and a classification video to be used, wherein the classification video to be used includes a plurality of video frames to be processed; and the target static image is an image different from the plurality of video frames to be processed; Inputting each to-be-processed video frame in the to-be-used classified video, the target static image and the fused image corresponding to each to-be-processed video frame into a pre-trained image fusion model in sequence to obtain a target fused image; wherein the fused image is obtained after fusion processing of the previous to-be-processed video frame, the fused image corresponding to the previous to-be-processed video frame and the target static image based on the image fusion model; Classify the target fusion image based on a pre-trained image classification model to obtain classification attribute features corresponding to the data to be processed; Based on the classification attribute characteristics, determining a target classification result corresponding to the target lesion; The step of sequentially inputting each to-be-processed video frame in the to-be-used classified video, the target static image, and the fused image corresponding to each to-be-processed video frame into a pre-trained image fusion model to obtain a target fused image includes: Inputting a first to-be-processed video frame in the to-be-used classified video and a target static image into an image fusion model, and obtaining a first fused image corresponding to the first to-be-processed video frame as a current fused image; Using the current fused image, the second video frame to be processed in the classification video to be used, and the target static image as inputs of the image fusion model, obtaining a second fused image corresponding to the second video frame to be processed, and updating the second fused image as the current fused image; The current fused image corresponding to the previous video frame, the current video frame to be processed and the target static image are sequentially used as inputs of the image fusion model to obtain a target fused image.

2. The method according to claim 1, characterized in that Before sequentially inputting each to-be-processed video frame in the to-be-used classified video, the target static image, and the fused image corresponding to each to-be-processed video frame into a pre-trained image fusion model to obtain the target fused image, the method includes: Input each to-be-processed video frame and target static image in the to-be-used classified video into the corresponding image preprocessing sub-model to obtain each processed video frame and target static image; wherein the image preprocessing sub-model is used to reduce the size of the to-be-processed video frame and target static image and increase the number of feature layers of the to-be-processed video frame and target static image.

3. The method according to claim 1, characterized in that The image fusion model includes a weight allocation sub-model, a target fusion sub-model and a feature processing sub-model. The current fused image, the second to-be-processed video frame in the to-be-used classified video and the target static image are used as inputs of the image fusion model to obtain a second fused image corresponding to the second to-be-processed video frame, and the second fused image is updated as the current fused image, including: Input the current fused image, the second video frame to be processed in the classification video to be used, and the target static image into the weight allocation sub-model to obtain fusion weight values ​​corresponding to the current fused image, the second video frame to be processed, and the target static image; Inputting the current fused image, the second video frame to be processed, the target static image and the corresponding fusion weight values ​​into the target fusion sub-model to obtain the current fused image to be used corresponding to the second video frame to be processed; The current fused image to be used is input into the feature processing sub-model to obtain a second fused image corresponding to the second video frame to be processed, and the second fused image is updated as the current fused image.

4. The method according to claim 3, characterized in that The weight allocation sub-model includes a feature preprocessing module, a feature mapping module, a normalization module and a weight allocation module. The current fused image, the second video frame to be processed in the classification video to be used, and the target static image are input into the weight allocation sub-model to obtain the fusion weight values ​​corresponding to the current fused image, the second video frame to be processed, and the target static image, including: Input the current fused image, the second video frame to be processed in the classification video to be used, and the target static image into the feature preprocessing module to obtain full-image feature data and spatial feature data corresponding to the current fused image, the second video frame to be processed, and the target static image; Inputting the full-image feature data and spatial feature data corresponding to the current fused image, the second video frame to be processed and the target static image into the feature mapping module to obtain the attention data to be processed corresponding to the current fused image, the second video frame to be processed and the target static image; Inputting the attention data to be processed corresponding to the current fused image, the second video frame to be processed and the target static image into the normalization module to obtain the target attention data corresponding to the current fused image, the second video frame to be processed and the target static image; Based on the weight allocation module, the target attention data corresponding to the current fused image, the second video frame to be processed and the target static image are processed to obtain the fusion weight values ​​corresponding to the current fused image, the second video frame to be processed and the target static image.

5. The method according to claim 3, characterized in that: The step of inputting the current fused image to be used into the feature processing sub-model to obtain a second fused image corresponding to the second video frame to be processed, and updating the second fused image to be the current fused image, further includes: Inputting the current fused image to be used into the feature processing sub-model to obtain a second fused image to be processed corresponding to the second video frame to be processed; If the second to-be-processed video frame is not the last to-be-processed video frame in the to-be-used classification video, upsampling is performed on the second to-be-processed fused image to obtain a second fused image, and the second fused image is updated as the current fused image.

6. The method according to claim 1, characterized in that The step of determining a target classification result corresponding to the target lesion based on the classification attribute feature includes: Based on the classification attribute characteristics, a target classification result corresponding to the target lesion is determined; wherein the target classification result includes normal condition, early condition, middle condition or late condition.

7. An image processing device, characterized in that: include: A module for acquiring data to be processed, used to acquire data to be processed of a target lesion; wherein the data to be processed includes a target static image corresponding to the target lesion and a classification video to be used, wherein the classification video to be used includes a plurality of video frames to be processed; and the target static image is an image different from the plurality of video frames to be processed; A target fused image acquisition module is used to sequentially input each to-be-processed video frame in the to-be-used classified video, the target static image and the fused image corresponding to each to-be-processed video frame into a pre-trained image fusion model to obtain a target fused image; wherein the fused image is obtained after fusion processing of the previous to-be-processed video frame, the fused image corresponding to the previous to-be-processed video frame and the target static image based on the image fusion model; A classification attribute feature acquisition module is used to classify the target fusion image based on a pre-trained image classification model to obtain classification attribute features corresponding to the data to be processed; A target classification result determination module, used to determine a target classification result corresponding to the target lesion based on the classification attribute characteristics; Wherein, the target fusion image acquisition module includes: a current fused image determining unit, configured to input a first to-be-processed video frame in the to-be-used classified video and a target static image into an image fusion model, and obtain a first fused image corresponding to the first to-be-processed video frame as a current fused image; a current fused image updating unit, configured to use the current fused image, the second video frame to be processed in the classification video to be used, and the target static image as inputs of the image fusion model, obtain a second fused image corresponding to the second video frame to be processed, and update the second fused image to the current fused image; The target fused image determination unit is used to sequentially execute the current fused image corresponding to the previous video frame, the current video frame to be processed and the target static image as inputs of the image fusion model to obtain the target fused image.

8. An electronic device, characterized in that: The device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image processing method as described in any one of claims 1 to 6 is implemented.

Images