Image classification method and device, equipment, storage medium and program product

By obtaining the feature information in the video clip and the feature information of the current frame image, and combining the three-dimensional and two-dimensional feature extraction networks, the problem of low accuracy in multi-structure classification in the digestive tract area is solved, and higher image classification accuracy is achieved.

CN119007055AActive Publication Date: 2024-11-22CHANGZHOU UNITED IMAGING HEALTHCARE SURGICAL TECH CO LTD +1
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202310551945.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-16
Publication Date
2024-11-22
Estimated Expiration
2043-05-16

AI Technical Summary

Technical Problem

In the prior art, the accuracy of image classification is not high, especially when the multi-structure classification of digestive tract sites, the classification accuracy is low.

Method used

By obtaining the video clips composed of the current frame image and its adjacent frame images, the feature extraction network is used to extract the feature information in the video clip and the current frame image, and performing fusion processing, combining the three-dimensional and two-dimensional feature extraction networks, and the fused Classify feature information to improve the accuracy of image classification.

Benefits of technology

It realizes a more accurate classification of digestive tract site images and improves classification accuracy. The specific value is 89.59%, which is better than traditional methods, such as Transformer for Fine-grained and Resnet series models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119007055A_ABST
    Figure CN119007055A_ABST
Patent Text Reader

Abstract

The invention relates to an image classification method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a current frame image of a current part; determining a video clip corresponding to the current frame image according to the current frame image; the video clip comprises a current frame image and an adjacent frame image corresponding to the current frame image; the method comprises the following steps: respectively extracting feature information in a video clip and a current frame image through a feature extraction network to obtain video feature information and image feature information, carrying out fusion processing on the video feature information and the image feature information through an image classification network, and classifying a fusion processing result to obtain a classification result of the current frame image. By adopting the method, the image classification accuracy can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image technology, and in particular to an image classification method, device, equipment, storage medium and program product. Background Art

[0002] With the development of artificial intelligence, the image classification process has attracted more and more attention. Image classification is an image processing method that distinguishes target areas of different categories based on different features reflected in image information.

[0003] Taking image classification as an example, images are classified according to their feature information. The more image categories obtained, the higher the accuracy of image classification.

[0004] However, in the related art, when classifying images, there is a problem that the image classification accuracy is not high. Summary of the invention

[0005] Based on this, it is necessary to provide an image classification method, device, equipment, storage medium and program product that can improve the accuracy of image classification in response to the above technical problems.

[0006] In a first aspect, the present application provides an image classification method, the method comprising:

[0007] Get the current frame image of the current part;

[0008] Determine, according to the current frame image, a video segment corresponding to the current frame image; the video segment includes the current frame image and adjacent frame images corresponding to the current frame image;

[0009] The feature extraction network extracts feature information from the video clip and the current frame image respectively to obtain video feature information and image feature information. The image classification network fuses the video feature information and the image feature information, and classifies the fusion result to obtain the classification result of the current frame image.

[0010] In one embodiment, the feature extraction network includes a three-dimensional feature extraction network and a two-dimensional feature extraction network; the feature extraction network is used to extract feature information in the video clip and the current frame image respectively to obtain video feature information and image feature information, including:

[0011] The feature information in the video clip is extracted through a three-dimensional feature extraction network to obtain video feature information; and the feature information of the current frame image is extracted through a two-dimensional feature extraction network to obtain image feature information.

[0012] In one embodiment, feature information in a video clip is extracted by a three-dimensional feature extraction network to obtain video feature information, including:

[0013] Extracting local feature information in the video clip through a three-dimensional feature extraction network to obtain local video feature information; and extracting global feature information in the video clip through a three-dimensional feature extraction network to obtain global video feature information;

[0014] The local video feature information and the global video feature information are superimposed to obtain the video feature information.

[0015] In one embodiment, feature information of the current frame image is extracted through a two-dimensional feature extraction network to obtain image feature information; including:

[0016] Extracting local feature information in the current frame image through a two-dimensional feature extraction network to obtain local image feature information; and extracting global feature information in the current frame image through a two-dimensional feature extraction network to obtain global image feature information;

[0017] The local image feature information and the global image feature information are superimposed to obtain the image feature information.

[0018] In one embodiment, the video feature information and the image feature information are fused to obtain a fusion result, including:

[0019] Extracting reference feature information corresponding to the current frame image from the video feature information;

[0020] The reference feature information is fused with the image feature information to obtain a fusion result.

[0021] In one embodiment, the reference feature information is fused with the image feature information to obtain a fusion result, including:

[0022] Obtaining the similarity between the reference feature information and the image feature information;

[0023] If the similarity is greater than a preset threshold, the reference feature information or the image feature information is used as the fusion result;

[0024] If the similarity is less than or equal to the preset threshold, a weighted sum is performed on the reference feature information and the image feature information, and the weighted sum result is used as the fusion result.

[0025] In a second aspect, the present application further provides an image classification device, the device comprising:

[0026] An acquisition module is used to acquire a current frame image of a current part;

[0027] A determination module, used to determine, based on the current frame image, a video segment corresponding to the current frame image; the video segment includes the current frame image and an adjacent frame image corresponding to the current frame image;

[0028] The processing module is used to input the video clip and the current frame image into a preset image classification network, and fuse the video feature information of the video clip and the image feature information of the current frame image through the image classification network to obtain the classification result of the current frame image.

[0029] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the content of any one of the image classification methods in the first aspect is implemented.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the content of any one of the image classification methods in the first aspect is implemented.

[0031] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the content of any one of the image classification methods in the first aspect when executed by a processor.

[0032] The above-mentioned image classification method, device, equipment, storage medium and program product obtain the current frame image of the current part, and determine the video clip corresponding to the current frame image according to the current frame image; the video clip includes the current frame image and the adjacent frame image corresponding to the current frame image, and the feature information in the video clip and the current frame image are respectively extracted by the feature extraction network to obtain the video feature information and the image feature information, and the image classification network fuses the video feature information and the image feature information, and classifies the result of the fusion processing to obtain the classification result of the current frame image. The method uses the feature extraction network to extract the feature information in the image and the video clip respectively. The process not only takes into account the feature information of the current frame, but also takes into account the features of the adjacent frames corresponding to the current frame, and fuses the two feature information. The information content contained in the fused features is more comprehensive. By classifying the results of the fusion processing, the classification result of the current frame image is made more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A diagram showing an application environment of an image classification method in one embodiment;

[0034] Figure 2 is a schematic diagram of a flow chart of an image classification method in one embodiment;

[0035] Figure 3A schematic diagram of a process of image classification in one embodiment;

[0036] Figure 4 is a schematic diagram of a flow chart of an image classification method in one embodiment;

[0037] Figure 5 A schematic diagram of extracting local feature information in one embodiment;

[0038] Figure 6 A schematic diagram of extracting global feature information in one embodiment;

[0039] Figure 7 A schematic diagram of a process for extracting video feature information in one embodiment;

[0040] Figure 8 is a schematic diagram of a flow chart of an image classification method in one embodiment;

[0041] Fig. 9 A schematic diagram of extracting local feature information in one embodiment;

[0042] Fig.10 A schematic diagram of extracting global feature information in one embodiment;

[0043] Fig.11 A schematic diagram of a process of extracting a current frame image in one embodiment;

[0044] Fig.12 is a schematic diagram of a flow chart of an image classification method in one embodiment;

[0045] Fig.13 is a schematic diagram of a flow chart of an image classification method in one embodiment;

[0046] Fig.14 is a schematic diagram of a flow chart of an image classification method in one embodiment;

[0047] Fig.15 FIG. 4 is a structural block diagram of an image classification device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] Before introducing the technical solution of the present application in detail, the background technology of the present application is first introduced.

[0050] In the process of classifying the image of the current part, a traditional neural network model can be used to classify different structures in the current part into one category to determine the specific structure contained in the current part.

[0051] Taking the current part as the digestive tract as an example, since the digestive tract includes many parts, the upper digestive tract alone includes the pharynx, esophagus, gastric antrum, pylorus, upper body, middle body, lower body, bone cutting and duodenum.

[0052] In the prior art, before the digestive tract parts were classified, it was first proposed to take photos of the gastric antrum, upper gastric body and lower gastric body in the digestive tract during upper gastrointestinal endoscopy (EGD). Four images are saved in each field of view, which refer to the anterior wall, lesser curvature of the stomach, posterior wall and greater curvature of the stomach. In addition, it is also necessary to take photos of the gastric fundus, stomach and gastric notch from a gyroscope perspective.

[0053] In the process of identifying the upper gastrointestinal tract, the traditional neural network method can be used to classify the partial structures of the digestive tract. Based on 34,513 pictures, each structure contains an average of 1,000-2,000 images, and its classification accuracy is 88.70% ± 0.23%, which can classify EGD images to a certain extent.

[0054] When the number of structures to be classified increases, the existing neural network has a low classification accuracy for multiple structures. In order to solve the problem of low classification accuracy, this application proposes an image classification method, which improves the neural network model to make it more accurate in classifying images.

[0055] It should be noted that this application not only uses images of multiple structures, but also medical videos corresponding to the images of multiple structures. The images and medical videos are all taken using the same equipment and operated by the same doctor. The image classification model in this scheme can detect images and medical videos in real time at a speed of 30.8FPS, so as to conduct more in-depth learning and improve the accuracy of image classification. This application uses different methods to classify multiple structures respectively. The classification accuracy is shown in Table 1. The classification accuracy of the neural architecture (Transformer for Fine-grained, TransFG) using fine-grained visual classification is 78.89%, the classification accuracy of the residual network (Residual Network 50, Resnet50) is 73.07%, the classification accuracy of Resnet 101 is 80.44%, and the classification accuracy of Resnet 152 is 80.59%. The classification accuracy of the image classification method provided in this application is 89.59%. It can be seen that the classification accuracy of the method of this application is the highest.

[0056] method Accuracy TransFG 78.89% Resnet 50 73.07% Resnet 101 80.44% Resnet 152 80.59% Image Classification Methods 89.59%

[0057] The image classification method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The computer device may include but is not limited to various terminals. For example, the terminal may be an image processor. The computer device includes a processor, a memory and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data in the image classification process. The network interface of the computer device is used to communicate with other external devices through a network connection, and the computer program is executed by the processor to implement an image classification method. Among them, the computer device can be implemented by an independent computer device or a computer device cluster composed of multiple computer devices.

[0058] In one embodiment, Figure 2 As shown, an image classification method is provided, which is applied to Figure 1 The computer device in the example is used to illustrate, including the following steps:

[0059] S201, obtaining a current frame image of a current part; the current frame image is collected by an endoscope device.

[0060] The current part may be any cavity of the human body that is connected to the outside world. The current medical image may be any medical image scanned by an endoscope device, and is not limited to the image scanned at the current moment.

[0061] Optionally, the computer device may search for historical images with the same identification information from an image database based on the identification information of the current part, and select any frame of image from the historical images as the current frame image of the current part. Optionally, the computer device may send a scanning instruction to a scanning device, and after receiving the scanning instruction, the scanning device scans the current part and sends the scanned image to the computer device, so that the computer device can obtain the current frame image. This embodiment does not limit the method for obtaining the current frame image of the current part.

[0062] S202, determining a video segment corresponding to the current frame image according to the current frame image; the video segment includes the current frame image and adjacent frame images corresponding to the current frame image.

[0063] For a frame image, the information it contains is only local information. It is impossible to fully and accurately classify the current frame image in a finer manner through local information. Therefore, it is also necessary to obtain the context information of multiple consecutive frames of images corresponding to the frame image to obtain the global information of the image, so as to more accurately classify the current frame image.

[0064] In this embodiment, after the computer device obtains the current frame image, it can search for multiple frame images adjacent to the sequence identifier from the image database according to the sequence identifier of the current frame image, and use the segment composed of the adjacent multiple frame images and the current frame image as the video segment. For example, the current frame image is the 4th frame, and the adjacent frame images can be the 2nd, 3rd, 5th, and 6th frames, and the 2nd to 6th frames are used as the video segment corresponding to the current frame image. It should be noted that the number of adjacent frame images can be any number, and the image segment is determined according to the number of adjacent frame images.

[0065] S203, extracting feature information from the video clip and the current frame image respectively through the feature extraction network to obtain video feature information and image feature information, and the image classification network fuses the video feature information and the image feature information, and classifies the fusion result to obtain the classification result of the current frame image.

[0066] In this embodiment, the computer device can extract video feature information from the video clip through a feature extraction network, and the video feature information is three-dimensional feature information on a time series, and extract image feature information from the current frame image through a feature extraction network, and the image feature information is two-dimensional feature information, and then extract the two-dimensional feature information corresponding to the image feature information in the three-dimensional feature information through an image classification network, and fuse the extracted two-dimensional feature information with the image feature information to obtain more information-rich fused feature information. After classifying the fused feature information, the classification result of the current frame image is more accurate. The classification result indicates that the current frame image belongs to a specific structure in the current part. For example, the classification result can be the center of the gastric angle, the anterior angle of the gastric angle, or the posterior angle of the gastric angle, etc.

[0067] Figure 3 This is a flowchart of image classification. Figure 3 In the method, the current frame image and the corresponding video clip are input into the image classification model, the feature information of the video clip is extracted by the feature extraction network in the image classification model to obtain the video feature information, and the feature information in the current frame image is extracted by the feature extraction network to obtain the image feature information, the video feature information and the image feature information are fused, and the fusion result is classified by the classifier to obtain the classification result of the current frame image.

[0068] It should be noted that both the feature extraction network and the image classification network belong to the image classification model. The image classification model is trained through a large number of sample images and sample video clips of the current part. Compared with the traditional method that only considers the features of a single frame image, the image classification model training process takes into account the features of the time series. Therefore, the trained image classification model is more accurate, and the classification process of the current frame image through the trained image classification model is also more accurate.

[0069] In the above-mentioned image classification method, the current frame image of the current part is obtained, and the video clip corresponding to the current frame image is determined according to the current frame image; the video clip includes the current frame image and the adjacent frame image corresponding to the current frame image, and the feature information in the video clip and the current frame image is extracted respectively by the feature extraction network to obtain the video feature information and the image feature information, and the image classification network fuses the video feature information and the image feature information, and classifies the result of the fusion processing to obtain the classification result of the current frame image. This method uses the feature extraction network to extract the feature information in the image and the video clip respectively. This process not only takes into account the feature information of the current frame, but also takes into account the features of the adjacent frames corresponding to the current frame, and fuses the two feature information. The information content contained in the fused features is more comprehensive. By classifying the results of the fusion processing, the classification result of the current frame image is more accurate.

[0070] On the basis of the above embodiment, in the case where the feature extraction network includes a three-dimensional feature extraction network and a two-dimensional feature extraction network, this embodiment is to Figure 2 The relevant contents of step S203 "extracting feature information from the video clip and the current frame image respectively through the feature extraction network to obtain video feature information and image feature information" are introduced and explained. The above step S203 may include the following contents: extracting feature information from the video clip through the three-dimensional feature extraction network to obtain video feature information; and extracting feature information from the current frame image through the two-dimensional feature extraction network to obtain image feature information.

[0071] In this embodiment, the computer device can input the video clip into the three-dimensional feature extraction network, extract the feature information in the video clip through the three-dimensional feature extraction network, and obtain the video feature information corresponding to the video clip. Also, the current frame image can be input into the two-dimensional feature extraction network, extract the feature information in the current frame image through the two-dimensional feature extraction network, and obtain the image feature information corresponding to the current frame image.

[0072] In the above-mentioned image classification method, feature information in the video clip is extracted through a three-dimensional feature extraction network to obtain video feature information; and feature information of the current frame image is extracted through a two-dimensional feature extraction network to obtain image feature information. This method extracts features of the video clip through a three-dimensional feature extraction network, and extracts features of the current frame image through a two-dimensional feature extraction network. Different feature extraction networks are used to extract corresponding feature information, so that the feature extraction process is more targeted and the extracted features are more accurate.

[0073] Based on the above embodiment, this embodiment introduces and explains the relevant contents of "extracting feature information from a video clip through a three-dimensional feature extraction network to obtain video feature information" in the above step. Figure 4 As shown, the above steps may include the following:

[0074] S301, extracting local feature information in the video clip through a three-dimensional feature extraction network to obtain local video feature information; and extracting global feature information in the video clip through a three-dimensional feature extraction network to obtain global video feature information.

[0075] Among them, the local feature information of a pixel refers to the feature information jointly determined by the features in eight directions adjacent to the pixel, and the global feature information of a pixel refers to the feature information determined by the features of all pixels.

[0076] In this embodiment, for any pixel, the computer device can use a three-dimensional feature extraction network to extract the feature information of the pixel, as well as extract the feature information of the eight directions adjacent to the pixel, and comprehensively consider the feature information of the pixel, the feature information of the eight adjacent directions, and the feature information of the adjacent frames corresponding to the pixel to obtain the local feature information of the pixel. In this way, the local feature information of all pixels can be extracted. It should be noted that when extracting local feature information, a time channel T is added to the traditional convolutional neural network (CNN), that is, a convolution operation is performed using a W*H*T convolution kernel. In this way, in addition to obtaining the local information of a frame of image, the context information between adjacent frame images can also be obtained. The local information and the context information are superimposed to obtain the local feature information in the video clip. Wherein, W represents the length of any frame of image in the video clip, and H represents the width of any frame of image in the video clip. Figure 5 A schematic diagram of extracting local feature information is shown, in which a pixel point whose feature needs to be extracted is selected from a video clip, and the pixel points adjacent to the pixel point and the corresponding pixel points in adjacent frame images are determined to obtain the local feature information of the pixel point.

[0077] Furthermore, the computer device can add a self-attention mechanism to CNN. The advantage of CNN is the ability to capture nearby local information, and the advantage of the self-attention mechanism is that it can better obtain the global information of the image. Therefore, combining CNN with the self-attention mechanism can combine local information with global information, better improve the feature expression ability of the model, and accurately obtain the global feature information in the video clip in the process of extracting features. Compared with the local feature extraction process, the extraction of global feature information is only to determine the information of a pixel through global feature information, so that the extracted features are more comprehensive and the information is richer. Among them, the processing process of CNN combined with the self-attention mechanism for video clips is: for any frame of the video clip, the features of each pixel in the image will be calculated with the features of all other pixels, and the weights of other pixels closer to the pixel account for a larger proportion, and the weights of other pixels farther from the pixel account for a smaller proportion. The feature information of the pixel can be obtained through the weight and the features of the pixel, and the global video feature information of the video clip can be obtained in this way. Figure 6 A schematic diagram of extracting global feature information is shown in FIG. 1 , where pixels whose features need to be extracted are selected from a video clip, and the feature information of all pixels is used to determine the global feature information of the pixel.

[0078] S302: Superimpose the local video feature information and the global video feature information to obtain video feature information.

[0079] In this embodiment, since the local video feature information focuses more on the extraction of local features, the extracted features are more accurate, but the features do not take into account the global features and have certain limitations. The global video feature information focuses more on the extraction of global features, and the extracted features take into account the global information comprehensively, but the global features do not take into account some more specific local feature information. Therefore, in order to make the obtained video feature information more comprehensive and accurate, after the computer device obtains the local video feature information and the global video feature information, the local video feature information and the global video feature information are superimposed, and only one of the repeated parts of the local video feature information and the global video feature information is taken, and for the different parts, all need to be retained, so that the obtained video feature information is more comprehensive and more accurate.

[0080] Figure 7 1 is a flow chart of extracting video feature information in an embodiment, wherein the local feature information and the global feature information in the video clip are extracted by using the W*H*T convolution kernel, and the two feature information are superimposed to obtain the video feature information.

[0081] In the above-mentioned image classification method, local feature information in the video clip is extracted through a three-dimensional feature extraction network to obtain local video feature information; and global feature information in the video clip is extracted through a three-dimensional feature extraction network to obtain global video feature information, and the local video feature information and the global video feature information are superimposed to obtain video feature information. This method extracts local video feature information and global video feature information in the video clip respectively through a three-dimensional feature extraction network, extracts different feature information from different angles, and superimposes the local video feature information and the global video feature information, so that the features included in the obtained video feature information are more comprehensive and more accurate.

[0082] Based on the above embodiment, this embodiment introduces and explains the relevant contents of "extracting feature information of the current frame image through a two-dimensional feature extraction network to obtain image feature information" in the above step. Figure 8 As shown, the above steps may include the following:

[0083] S401, extracting local feature information in the current frame image through a two-dimensional feature extraction network to obtain local image feature information; and extracting global feature information in the current frame image through a two-dimensional feature extraction network to obtain global image feature information.

[0084] In this embodiment, for any pixel point, the computer device can use a two-dimensional feature extraction network to extract the feature information of the pixel point, as well as extract the feature information of the eight directions adjacent to the pixel point, and comprehensively consider the feature information of the pixel point, the feature information of the eight adjacent directions, and the feature information of the adjacent frames corresponding to the pixel point to obtain the local feature information of the pixel point. In this way, the local feature information of all pixels can be extracted. It should be noted that when extracting local feature information, a convolution operation is performed using a W*H convolution kernel through a traditional CNN, so that the local feature information in the current frame image can be obtained. Fig. 9 A schematic diagram for extracting local feature information is shown, in which a pixel point whose feature needs to be extracted is selected from the current frame image, and the pixel points adjacent to the pixel point and the corresponding pixel points in the adjacent frame image are determined to obtain the local feature information of the pixel point.

[0085] Furthermore, similar to extracting global information from video clips, a self-attention mechanism can be added to the traditional CNN algorithm to accurately obtain the global feature information in the current frame image during the feature extraction process. Fig.10 A schematic diagram of extracting global feature information is shown in FIG. 1 , where a pixel point whose features need to be extracted is selected from the current frame image, and the global feature information of the pixel point is determined using the feature information of all the pixels.

[0086] S402: Superimpose the local image feature information and the global image feature information to obtain image feature information.

[0087] In this embodiment, after the computer device obtains the local video feature information and the global video feature information, the local video feature information and the global video feature information are superimposed. Only one of the repeated parts of the local video feature information and the global video feature information is taken, and all the different parts need to be retained, so that the obtained video feature information is more comprehensive and more accurate.

[0088] Fig.11 It is a schematic diagram of the process of extracting the current frame image in an embodiment, extracting the local feature information and the global feature information in the current frame image through the W*H convolution kernel, and superimposing the two feature information to obtain the video feature information.

[0089] In the above-mentioned image classification method, local feature information in the current frame image is extracted through a two-dimensional feature extraction network to obtain local image feature information; and global feature information in the current frame image is extracted through a two-dimensional feature extraction network to obtain global image feature information, and the local image feature information and the global image feature information are superimposed to obtain image feature information. This method extracts local image feature information and global image feature information in the current frame image respectively through a two-dimensional feature extraction network, extracts different feature information from different angles, and superimposes the local image feature information and the global image feature information, so that the features included in the obtained image feature information are more comprehensive and more accurate.

[0090] On the basis of the above embodiment, this embodiment introduces and explains the relevant contents of "fusing the video feature information and the image feature information to obtain a fusion result" in the above step S203.

[0091] like Fig.12 As shown, the above step S203 may include the following contents:

[0092] S501, extracting reference feature information corresponding to the current frame image from the video feature information.

[0093] In this embodiment, since the video feature information is three-dimensional feature information, the three-dimensional feature information includes feature information corresponding to multiple frames of images, and the image feature information is two-dimensional feature information, in order to take into account the feature information of the video clip, it is necessary to extract the feature information corresponding to the current frame image from the video feature information, and use the feature information as reference feature information. For example, the video clip includes 7 frames of images, the current frame image belongs to the 4th frame of all frames in the video clip, and the current frame image and the 4th frame image in the video clip are the same frame image in the time dimension. Since the video feature information is composed of feature information of 7 frames of images, the feature information of the 4th frame image in the video clip takes into account the feature information of adjacent frames, so the feature information of the 4th frame image in the video clip may be different from the feature information of the current frame image. Therefore, it is necessary to extract the reference feature information corresponding to the current frame image from the video feature information, and the reference feature information is the feature information corresponding to the 4th frame image in the video feature information.

[0094] S502: Fuse the reference feature information with the image feature information to obtain a fusion result.

[0095] In this embodiment, in order to obtain more comprehensive information in the current frame image, it is necessary to fuse the reference feature information with the image feature information to obtain more comprehensive and accurate feature information. Therefore, the computer device can fuse the reference feature information with the image feature information, fix the parts with the same feature information, and fuse the parts with different features in a weighted summation manner to obtain the fusion result of the reference feature information and the image feature information. The fusion result can more comprehensively and accurately reflect the feature information of the current frame image.

[0096] It should be noted that the two-dimensional feature extraction network and the three-dimensional feature extraction network adopt the idea of ​​teacher-student network, with the two-dimensional feature extraction network as the teacher network and the three-dimensional feature extraction network as the student network. The feature information output by the teacher network is weightedly fused with the feature information output by the student network to output the final fusion result.

[0097] In the above-mentioned image classification method, reference feature information is extracted from the video feature information; the reference feature information and the image feature information are information of the same feature, and the reference feature information and the image feature information are fused to obtain a fusion result. This method extracts reference feature information of the same feature as the image feature information from the video feature information, and fuses the reference information with the image feature information. The obtained fusion result includes not only the local image features and global image features of the current frame image, but also the features of the adjacent frame images, thereby obtaining a more comprehensive fusion feature.

[0098] On the basis of the above embodiment, this embodiment introduces and explains the relevant contents of "fusing the video feature information and the image feature information to obtain a fusion result" in the above step S502.

[0099] like Fig.13 As shown, the above step S502 may include the following contents:

[0100] S601, obtaining the similarity between the reference feature information and the image feature information.

[0101] In this embodiment, the computer device can determine whether the reference feature information of each pixel is the same as the corresponding image feature information, obtain the number of pixels with the same feature information, calculate the ratio of the number of pixels with the same feature information to the total number of pixels, and determine the ratio as the similarity between the reference feature information and the image feature information. For example, if the total number of pixels is 100 and the number of pixels with the same feature is 80, the similarity between the reference feature information and the image feature information is 0.8.

[0102] S602: If the similarity is greater than a preset threshold, the reference feature information or the image feature information is used as a fusion result.

[0103] In this embodiment, the computer device can compare the similarity with a preset threshold. If the similarity is greater than the preset threshold, it means that the reference feature information is basically the same as the image feature information, and only a few features are different. In this case, the reference feature information can be used as the fusion result, and the reference feature information can be used as the fusion result.

[0104] S603: If the similarity is less than or equal to a preset threshold, weighted sum is performed on the reference feature information and the image feature information, and the weighted sum result is used as a fusion result.

[0105] In this embodiment, when the similarity is less than or equal to the preset threshold, it means that the reference feature information and the image feature information are partially different. Therefore, it is necessary to perform weighted summation on the reference feature information and the image feature information, and use the weighted summation result as the fusion result. The weighting coefficients of the reference feature information and the image feature information can be obtained based on historical experience. For example, the weighting coefficient of the reference feature information can be 0.4, and the weighting coefficient of the image feature information can be 0.6.

[0106] In the above-mentioned image classification method, the similarity between the reference feature information and the image feature information is obtained. If the similarity is greater than a preset threshold, the reference feature information or the image feature information is used as the fusion result. If the similarity is less than or equal to the preset threshold, the reference feature information and the image feature information are weighted and summed, and the weighted sum result is used as the fusion result. By obtaining the similarity between the reference feature information and the image feature information and comparing the similarity with the preset threshold, the method can obtain the fusion result in different ways, so that the obtained fusion result is more accurate.

[0107] In one embodiment, the image classification method is described in detail below. Fig.14 As shown, the method may include:

[0108] S701, obtaining a current frame image of a current part;

[0109] S702, determining a video segment corresponding to the current frame image according to the current frame image;

[0110] S703, extracting local feature information in the video clip through a three-dimensional feature extraction network to obtain local video feature information; and extracting global feature information in the video clip through a three-dimensional feature extraction network to obtain global video feature information;

[0111] S704, superimposing the local video feature information and the global video feature information to obtain video feature information;

[0112] S705, extracting local feature information in the current frame image through a two-dimensional feature extraction network to obtain local image feature information; and extracting global feature information in the current frame image through a two-dimensional feature extraction network to obtain global image feature information;

[0113] S706, superimposing the local image feature information and the global image feature information to obtain image feature information;

[0114] S707, extracting reference feature information corresponding to the current frame image from the video feature information;

[0115] S708, obtaining the similarity between the reference feature information and the image feature information;

[0116] S709, if the similarity is greater than a preset threshold, the reference feature information or the image feature information is used as a fusion result;

[0117] S710: If the similarity is less than or equal to a preset threshold, weighted sum is performed on the reference feature information and the image feature information, and the weighted sum result is used as a fusion result.

[0118] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0119] Based on the same inventive concept, the embodiment of the present application also provides an image classification device for implementing the above-mentioned image classification method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image classification device embodiments provided below can refer to the limitations on the image classification method above, and will not be repeated here.

[0120] In one embodiment, Fig.15 As shown, an image classification device is provided, comprising: an acquisition module 11, a determination module 12 and a processing module 13, wherein:

[0121] An acquisition module 11 is used to acquire a current frame image of a current part;

[0122] The determination module 12 is used to determine the video segment corresponding to the current frame image according to the current frame image; the video segment includes the current frame image and the adjacent frame image corresponding to the current frame image;

[0123] The processing module 13 is used to input the video clip and the current frame image into a preset image classification network, and fuse the video feature information of the video clip and the image feature information of the current frame image through the image classification network to obtain a classification result of the current frame image.

[0124] In one embodiment, the classification model includes a first extraction unit, wherein:

[0125] The first extraction unit is used to extract feature information in the video clip through a three-dimensional feature extraction network to obtain video feature information; and to extract feature information of the current frame image through a two-dimensional feature extraction network to obtain image feature information.

[0126] In one embodiment, the first extraction unit is further used to extract local feature information in the video clip through a three-dimensional feature extraction network to obtain local video feature information; and to extract global feature information in the video clip through a three-dimensional feature extraction network to obtain global video feature information; and to superimpose the local video feature information and the global video feature information to obtain video feature information.

[0127] In one embodiment, the first extraction unit is also used to extract local feature information in the current frame image through a two-dimensional feature extraction network to obtain local image feature information; and to extract global feature information in the current frame image through a two-dimensional feature extraction network to obtain global image feature information; and to superimpose the local image feature information and the global image feature information to obtain image feature information.

[0128] In one embodiment, the processing module further includes: a second extraction unit and a fusion unit, wherein:

[0129] A second extraction unit, used to extract reference feature information corresponding to the current frame image from the video feature information;

[0130] The fusion unit is used to fuse the reference feature information with the image feature information to obtain a fusion result.

[0131] In one embodiment, the fusion unit is also used to obtain the similarity between the reference feature information and the image feature information; if the similarity is greater than a preset threshold, the reference feature information or the image feature information is used as the fusion result; if the similarity is less than or equal to the preset threshold, the reference feature information and the image feature information are weightedly summed, and the weighted sum result is used as the fusion result.

[0132] Each module in the above-mentioned image classification device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0133] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the content of any one of the above-mentioned image classification method embodiments is implemented.

[0134] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the content of any one of the above-mentioned image classification method embodiments is implemented.

[0135] In one embodiment, a computer program product is provided, including a computer program, which implements the content of any one of the above-mentioned image classification method embodiments when executed by a processor.

[0136] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0137] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0138] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image classification method, It is characterized in that The method comprises: Acquire a current frame image of a current part; the current frame image is collected by an endoscope device; Determine, according to the current frame image, a video segment corresponding to the current frame image; the video segment includes the current frame image and adjacent frame images corresponding to the current frame image; The feature extraction network extracts feature information from the video clip and the current frame image respectively to obtain video feature information and image feature information. The image classification network fuses the video feature information and the image feature information, and classifies the fusion result to obtain the classification result of the current frame image.

2. The method according to claim 1, It is characterized in that The feature extraction network includes a three-dimensional feature extraction network and a two-dimensional feature extraction network; the feature extraction network is used to extract feature information from the video clip and the current frame image respectively to obtain video feature information and image feature information, including: The feature information in the video clip is extracted by the three-dimensional feature extraction network to obtain the video feature information; and the feature information of the current frame image is extracted by the two-dimensional feature extraction network to obtain the image feature information.

3. The method according to claim 2, It is characterized in that The extracting feature information from the video clip by the three-dimensional feature extraction network to obtain the video feature information includes: Extracting local feature information in the video clip through the three-dimensional feature extraction network to obtain local video feature information; and extracting global feature information in the video clip through the three-dimensional feature extraction network to obtain global video feature information; The local video feature information and the global video feature information are superimposed to obtain the video feature information.

4. The method according to claim 2, It is characterized in that The extracting feature information of the current frame image through the two-dimensional feature extraction network to obtain the image feature information comprises: Extracting local feature information in the current frame image through the two-dimensional feature extraction network to obtain local image feature information; and extracting global feature information in the current frame image through the two-dimensional feature extraction network to obtain global image feature information; The local image feature information and the global image feature information are superimposed to obtain the image feature information.

5. The method according to claim 1, It is characterized in that The fusing the video feature information and the image feature information to obtain a fusion result includes: Extracting reference feature information corresponding to the current frame image from the video feature information; The reference feature information is fused with the image feature information to obtain the fusion result.

6. The method according to claim 5, It is characterized in that The fusing the reference feature information with the image feature information to obtain the fusion result includes: Acquire the similarity between the reference feature information and the image feature information; If the similarity is greater than a preset threshold, taking the reference feature information or the image feature information as the fusion result; If the similarity is less than or equal to a preset threshold, a weighted sum is performed on the reference feature information and the image feature information, and the weighted sum result is used as the fusion result.

7. An image classification device, It is characterized in that The device comprises: An acquisition module, used to acquire a current frame image of a current part; the current frame image is acquired by an endoscope device; A determination module, configured to determine, based on the current frame image, a video segment corresponding to the current frame image; the video segment includes the current frame image and adjacent frame images corresponding to the current frame image; A processing module is used to input the video clip and the current frame image into a preset image classification network, and fuse the video feature information of the video clip and the image feature information of the current frame image through the image classification network to obtain a classification result of the current frame image.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Network violent video identification method

    CN103218608A

  • Pedestrian re-identification method based on spatial and temporal features

    CN111680560A

  • Video identification method and device based on adaptive reasoning

    CN114241360A

  • Video-based focus classification method and device, electronic equipment and medium

    CN114663372A

  • Breast lesion attribute prediction method and device based on ultrasonic video

    CN114842239A