Target classification method and device based on image recognition and storage medium

Abstract

The invention relates to the technical field of data processing, in particular to a target classification method and device based on image recognition and a storage medium. Comprising the following steps: acquiring at least two frames of to-be-recognized images including the cardiac muscle of the same to-be-detected subject; determining time sequence features and spatial features of the to-be-recognized image; the time sequence features are used for representing the time sequence of at least two frames of to-be-recognized images, and the spatial features are used for representing the myocardial distribution condition in each frame of to-be-recognized image; and determining the probability that the at least two frames of to-be-identified images belong to the target classification according to the time sequence features and the spatial features. The embodiment of the invention is used for solving the problem that the current image classification recognition mode consumes long time.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device and storage medium for target classification based on image recognition. Background Art

[0002] Cardiac Cinematic Magnetic Resonance Imaging (Cine-MRI) technology can provide dynamic images of the heart during movement, and imaging omics technology can extract feature information from images. Therefore, imaging omics technology can be used to extract image features from dynamic images collected by cardiac MRI film technology and classify and identify dynamic images. However, imaging omics technology usually requires manual delineation of specific feature parts, and a dynamic image collected usually corresponds to at least dozens of frames of images, so this classification and identification method is time-consuming. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a target classification method, device and storage medium based on image recognition, which can save classification and recognition time.

[0004] In a first aspect, the present application provides a target classification method based on image recognition, comprising: acquiring at least two frames of images to be identified including the myocardium of the same subject to be detected; determining the timing characteristics and spatial characteristics of the images to be identified; the timing characteristics are used to characterize the time sequence of at least two frames of images to be identified, and the spatial characteristics are used to characterize the distribution of the myocardium in each frame of the image to be identified; based on the timing characteristics and the spatial characteristics, determining the probability that at least two frames of images to be identified belong to the target classification.

[0005] In some embodiments, determining the temporal features and spatial features of an image to be identified includes: inputting at least two frames of images to be identified into a spatiotemporal recognition model to obtain the temporal features and spatial features of the image to be identified; the spatiotemporal recognition model is generated based on training of multiple segments of a first training set of dynamic images, and the first training set of dynamic images includes a myocardial image of a first subject.

[0006] In some embodiments, before determining the temporal features and spatial features of the image to be identified, the method also includes: acquiring multiple segments of first training set dynamic images; a segment of the first training set dynamic image includes at least two frames of historical images of the myocardium of a first subject; for each segment of the first training set dynamic image, processing operations are performed on the historical images of the first training set dynamic image to obtain a disordered image and a missing image corresponding to each segment of the first training set dynamic image; wherein the processing operations include: shuffling the time sequence of the historical images in the first training set dynamic image to obtain a disordered image; performing random position masking on each frame of the historical image in the first training set dynamic image to obtain a missing image; using the disordered image and the missing image corresponding to the first training set dynamic image as input, and the time sequence and historical images of the historical images in the first training set dynamic image as output targets, and training to obtain a spatiotemporal recognition model.

[0007] In some embodiments, before processing the historical images of the first training set dynamic images, the method also includes: preprocessing each segment of the first training set dynamic images according to preset rules to obtain historical images corresponding to each segment of the first training set dynamic images; the preset rules are used to limit the specifications of the historical images corresponding to the first training set dynamic images.

[0008] In some embodiments, the probability that the entire myocardium in the at least two frames of images to be identified belongs to the target category is determined as the probability that the at least two frames of images to be identified belong to the target category; or, the probability that each myocardial segment in the at least two frames of images to be identified belongs to the target category is determined as the probability that the at least two frames of images to be identified belong to the target category.

[0009] In some embodiments, the probability that at least two frames of images to be identified belong to the target category is determined based on the temporal features and the spatial features, including: inputting the temporal features and the spatial features into a multi-label classification model to obtain the probability that at least two frames of images to be identified belong to the target category; the multi-label classification model is generated by training based on the temporal features of multiple segments of second training set dynamic images, the spatial features of multiple segments of second training set dynamic images, and the classification labels of each myocardial segment corresponding to each segment of the second training set dynamic images; or, the multi-label classification model is generated by training based on the temporal features of multiple segments of second training set dynamic images, the spatial features of multiple segments of second training set dynamic images, and the classification labels corresponding to the entire myocardium in each segment of the second training set dynamic images; the second training set dynamic images include myocardial images of the second subject.

[0010] In the second aspect, the present application provides a target classification device based on image recognition, including: an acquisition module, used to acquire at least two frames of images to be identified including the myocardium of the same subject to be detected; a determination module, used to determine the timing characteristics and spatial characteristics of the images to be identified; the timing characteristics are used to characterize the time sequence of at least two frames of images to be identified, and the spatial characteristics are used to characterize the distribution of the myocardium in each frame of the image to be identified; the recognition module is used to determine the probability that at least two frames of images to be identified belong to the target classification based on the timing characteristics and spatial characteristics.

[0011] In some embodiments, the determination module is specifically used to input at least two frames of images to be identified into a spatiotemporal recognition model to obtain temporal features and spatial features of the images to be identified; the spatiotemporal recognition model is generated based on training of multiple segments of first training set dynamic images, and the first training set dynamic images include myocardial images of the first subject.

[0012] In some embodiments, the target classification device based on image recognition also includes a processing module and a training module; the acquisition module is also used to acquire multiple segments of first training set dynamic images before determining the temporal features and spatial features of the image to be identified; a segment of the first training set dynamic image includes at least two frames of historical images of the myocardium of a first subject; the processing module is used to perform processing operations on the historical images of the first training set dynamic images for each segment of the first training set dynamic images, so as to obtain disordered images and missing images corresponding to each segment of the first training set dynamic images; wherein the processing operations include: shuffling the time sequence of the historical images in the first training set dynamic images to obtain disordered images; performing random position masking processing on each frame of the historical images in the first training set dynamic images to obtain missing images; the training module is used to take the disordered images and missing images corresponding to the first training set dynamic images as input, and the time sequence and historical images of the historical images in the first training set dynamic images as output targets, and train to obtain a spatiotemporal recognition model.

[0013] In some embodiments, the processing module is further used to pre-process each segment of the first training set dynamic image according to preset rules to obtain a historical image corresponding to each segment of the first training set dynamic image; the preset rules are used to limit the specifications of the historical images corresponding to the first training set dynamic images.

[0014] In some embodiments, the recognition module is specifically used to determine the probability that the entire myocardium in the at least two frames of images to be recognized belongs to the target category as the probability that the at least two frames of images to be recognized belong to the target category; or, to determine the probability that each myocardial segment in the at least two frames of images to be recognized belongs to the target category as the probability that the at least two frames of images to be recognized belong to the target category.

[0015] In some embodiments, the recognition module is specifically used to input the temporal features and the spatial features into a multi-label classification model to obtain the probability that at least two frames of images to be identified belong to the target classification; the multi-label classification model is generated by training based on the temporal features of multiple segments of second training set dynamic images, the spatial features of multiple segments of second training set dynamic images, and the classification labels of each myocardial segment corresponding to each segment of second training set dynamic images; or, the multi-label classification model is generated by training based on the temporal features of multiple segments of second training set dynamic images, the spatial features of multiple segments of second training set dynamic images, and the classification labels corresponding to the entire myocardium in each segment of second training set dynamic images; the second training set dynamic images include the myocardial images of the second subject.

[0016] In a third aspect, the present application provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, an image recognition-based target classification method as in any embodiment of the first aspect is implemented.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the target classification method based on image recognition in any embodiment of the first aspect.

[0018] In a fifth aspect, the present application provides a computer program product, comprising: when the computer program product runs on a computer, the computer implements the target classification method based on image recognition as in any embodiment of the first aspect.

[0019] The technical solution provided by the present application has the following advantages compared with the prior art: First, at least two frames of images to be identified including the myocardium of the same subject to be detected are obtained, and the temporal features and spatial features of the images to be identified are determined. Among them, the temporal features are used to characterize the time sequence of at least two frames of images to be identified, and the spatial features are used to characterize the distribution of the myocardium in each frame of the image to be identified. Then, based on the temporal features and the spatial features, the probability that at least two frames of images to be identified belong to the target classification is determined. In this way, when performing target classification of the images to be identified, the temporal features and the spatial features of the images to be identified can be determined first, and then the probability that at least two frames of images to be identified belong to the target classification can be directly determined based on the temporal features and the spatial features, thereby avoiding the problem of manual participation in the process of determining the myocardial classification in the images to be identified, saving recognition time, and improving the accuracy of the recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Figure 1 One of the schematic diagrams of application scenarios of the target classification method based on image recognition provided in an embodiment of the present application;

[0023] Figure 2 One of the flowcharts of the target classification method based on image recognition provided in the embodiment of the present application;

[0024] Figure 3 The second flowchart of the target classification method based on image recognition provided in the embodiment of the present application;

[0025] Figure 4 The third flowchart of the target classification method based on image recognition provided in the embodiment of the present application;

[0026] Figure 5 A second schematic diagram of an application scenario of the target classification method based on image recognition provided in an embodiment of the present application;

[0027] Figure 6 A fourth flowchart of a target classification method based on image recognition provided in an embodiment of the present application;

[0028] Figure 7 A fifth flowchart of a target classification method based on image recognition provided in an embodiment of the present application;

[0029] Figure 8 Flow chart six of the target classification method based on image recognition provided in the embodiment of the present application;

[0030] Fig. 9 The third schematic diagram of an application scenario of the target classification method based on image recognition provided in an embodiment of the present application;

[0031] Fig.10 One of the structural schematic diagrams of the target classification device based on image recognition provided in an embodiment of the present application;

[0032] Fig.11 The second structural diagram of the target classification device based on image recognition provided in an embodiment of the present application;

[0033] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0035] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0036] Figure 1 A schematic diagram of the scene architecture of the target classification method based on image recognition provided in the embodiment of the present application. Figure 1 As shown, the scenario architecture provided by the embodiment of the present application includes: a server 100 and an electronic device 200.

[0037] The electronic device 200 provided in the embodiment of the present application may have various implementation forms, for example, it may be a mobile phone, a personal computer (PC), a smart TV, a laser projection device, a TV, a monitor, a wearable device, a vehicle-mounted device, an electronic table, etc.

[0038] In some embodiments, when the electronic device 200 receives the target classification instruction based on image recognition, it can communicate data with the server 100. The electronic device 200 can be allowed to communicate with the server 100 through a local area network (LAN) or a wireless local area network (WLAN).

[0039] The server 100 may be a server that provides various services, such as a server that provides support for the image to be recognized obtained by the electronic device 200. The server may perform feature determination, target classification and recognition on the received image to be recognized, and feed back the processing result to the electronic device 200. The server 100 may be a server cluster or multiple server clusters, and may include one or more types of servers.

[0040] It should be noted that the target classification method based on image recognition provided in the embodiment of the present application can be executed by the server 100, or by the electronic device 200, or by both the server 100 and the electronic device 200, and the present application does not limit this.

[0041] The target classification method based on image recognition provided in the embodiment of the present application can be executed by a target classification device based on image recognition, and the target classification device based on image recognition can be hardware or software. When the target classification device based on image recognition is hardware, it can be various electronic devices with a target classification function based on image recognition, including but not limited to mobile phones, computers, computers, tablet computers, televisions, smart TVs, vehicle-mounted equipment, smart vehicles, laser projection equipment, monitors, electronic whiteboards (electronic bulletin board), electronic desktops (electronic table), etc. When the target classification device based on image recognition is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules, or it can be implemented as a single software or software module. It is not specifically limited here.

[0042] Figure 2 A flow chart of a target classification method based on image recognition provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the target classification method based on image recognition may include the following steps.

[0043] S11, acquiring at least two frames of images to be identified including the myocardium of the same subject to be detected.

[0044] Specifically, the method of obtaining the image to be identified can be to collect the video image corresponding to the myocardium of the subject to be detected through relevant medical equipment, such as an MRI scanner, a computed tomography (CT) scanner, etc. Then, the video image containing the myocardium of the subject is collected at a preset frequency to obtain multiple frames of images to be identified.

[0045] S12: Determine the temporal features and spatial features of the image to be identified.

[0046] Among them, the temporal feature is used to characterize the time sequence of at least two frames of images to be identified, that is, the playback order of each frame of images to be identified in the corresponding video image. The spatial feature is used to characterize the distribution of myocardium in each frame of images to be identified, that is, the position and structure of myocardium in each frame of images to be identified.

[0047] In some embodiments, the method for determining the temporal features and spatial features of the image to be identified can be to use a pre-trained spatiotemporal recognition model to determine the temporal features and spatial features of the image to be identified; or a temporal algorithm can be used to calculate the motion vectors in different images to be identified, capture the direction and speed of the object's movement, and thus identify the temporal features between the images to be identified. For example, the temporal algorithm can be Lucas-Kanade, Farneback, etc. in the optical flow method; the spatial features of the image to be identified are identified by an image recognition algorithm, for example, R-CNN, Fast R-CNN, Faster R-CNN, YOLO, etc.

[0048] S13. Determine the probability that at least two frames of to-be-recognized images belong to the target category according to the temporal features and the spatial features.

[0049] Among them, target classification is used to characterize the myocardial state of the subject to be tested, for example, myocardial ischemia state, myocardial abnormal shape state, etc.

[0050] Specifically, the method for determining the probability that at least two frames of images to be identified belong to the target category based on temporal features and spatial features can be to use a pre-trained recognition model to identify the probability that at least two frames of images to be identified belong to the target category based on temporal features and spatial features.

[0051] In the above scheme, first, at least two frames of images to be identified including the myocardium of the same subject to be detected are obtained, and the temporal features and spatial features of the images to be identified are determined. Among them, the temporal features are used to characterize the time sequence of at least two frames of images to be identified, and the spatial features are used to characterize the distribution of the myocardium in each frame of the image to be identified. Then, based on the temporal features and the spatial features, the probability that at least two frames of images to be identified belong to the target classification is determined. In this way, when performing target classification of the images to be identified, the temporal features and the spatial features of the images to be identified can be determined first, and then the probability that at least two frames of images to be identified belong to the target classification can be directly determined based on the temporal features and the spatial features, thereby avoiding the problem of manual participation in the process of determining the myocardial classification in the images to be identified, saving recognition time, and improving the accuracy of the recognition results.

[0052] In some embodiments, Figure 3 As shown, the method of determining the temporal features and spatial features of the image to be identified may include the following steps:

[0053] S121, inputting at least two frames of images to be recognized into a spatiotemporal recognition model to obtain temporal features and spatial features of the images to be recognized.

[0054] The spatiotemporal recognition model is generated by training based on multiple segments of first training set dynamic images, and the first training set dynamic images include myocardial images of the first subject.

[0055] Specifically, Figure 4 As shown, the method of obtaining the spatiotemporal recognition model by training multiple first training set dynamic images may include the following steps:

[0056] S21, obtaining multiple segments of first training set dynamic images.

[0057] Wherein, a first training set of dynamic images includes at least two frames of historical images of the myocardium of a first subject.

[0058] Specifically, the method of obtaining multiple segments of dynamic images of the first training set can be to use relevant medical equipment to collect video images corresponding to the myocardiums of different first subjects in real time to obtain multiple segments of dynamic images of the first training set; or it can be to directly retrieve video images corresponding to the myocardiums of different first subjects from different data acquisition centers to obtain multiple segments of dynamic images of the first training set; that is, this application does not limit the source of the dynamic images of the first training set.

[0059] S22 . For each segment of the first training set dynamic image, a processing operation is performed on the historical images of the first training set dynamic image to obtain a disordered image and a missing image corresponding to each segment of the first training set dynamic image.

[0060] The processing operation includes: disrupting the time sequence of the historical images in the first training set of dynamic images to obtain disordered images; and masking each frame of the historical images in the first training set of dynamic images at a random position to obtain missing images.

[0061] Specifically, the historical images in the first training set dynamic image are processed in a time order to obtain a disordered image by processing the historical images through a disorder algorithm. For example, the Fisher-Yates shuffle algorithm, the random interval extraction algorithm, the time window shuffle algorithm, etc. The historical images in the first training set dynamic image are processed in a random position masking process to obtain a missing image by using a masking algorithm to mask each historical image in a random position to obtain a missing image. For example, the random rectangle masking algorithm, the random masking grid algorithm, etc.

[0062] S23, taking the disordered images and missing images corresponding to the dynamic images of the first training set as input, and the time sequence of the historical images and the historical images in the dynamic images of the first training set as output targets, and training to obtain a spatiotemporal recognition model.

[0063] For example, Figure 5As shown, the disordered images and missing images corresponding to the dynamic images of the first training set are used as input, the time sequence of the historical images in the dynamic images of the first training set and the historical images are used as output targets, and the parameters of the model are adjusted according to the Kendall's Taul loss until the Kendall's Taul loss reaches a preset range to obtain a spatiotemporal recognition model. The Kendall's Taul loss includes the Kendall's Taul loss between the output image of the model and the historical image, and the Kendall's Taul loss between the time sequence of the model output image and the time sequence of the historical image.

[0064] In the above scheme, the spatiotemporal recognition model can learn the temporal characteristics and spatial characteristics of the dynamic images corresponding to the myocardium of the subject based on the first training set dynamic images corresponding to the myocardium of the first subject. In this way, when processing the image to be recognized corresponding to the myocardium of the subject to be detected, the spatiotemporal recognition model can be used instead of manual delineation to automatically recognize the time sequence and spatial structure of the image to be recognized, which greatly saves the processing time of the entire recognition process and avoids the artificial errors that may be introduced during manual processing, thereby improving the recognition accuracy.

[0065] In some embodiments, before the historical images of the first training set dynamic images are processed, the target classification method based on image recognition further includes: preprocessing each segment of the first training set dynamic images according to preset rules to obtain the historical images corresponding to each segment of the first training set dynamic images. The preset rules are default rules, or rules formulated by relevant personnel according to actual conditions, which are used to limit the specifications of the historical images corresponding to the first training set dynamic images. For example, the preset rules may include denoising, sampling, unifying scales, formats, and number of frames of the first training set dynamic images to obtain standardized historical images.

[0066] In the above scheme, the specifications of historical images are unified before training the spatiotemporal recognition model, which avoids the influence of the specifications of historical images on the accuracy of the spatiotemporal recognition model and indirectly improves the accuracy of determining the temporal features and spatial features of the images to be recognized.

[0067] In some embodiments, the method for determining the probability that at least two frames of images to be identified belong to the target category may be to determine the probability that the entire myocardium in at least two frames of images to be identified belongs to the target category as the probability that at least two frames of images to be identified belong to the target category; or it may be to determine the probability that each myocardial segment in at least two frames of images to be identified belongs to the target category as the probability that at least two frames of images to be identified belong to the target category. Wherein, when determining the probability that each myocardial segment in at least two frames of images to be identified belongs to the target category as the probability that at least two frames of images to be identified belong to the target category, the probability that at least two frames of images to be identified belong to the target category also includes the probabilities corresponding to multiple segments, and the probability of one segment corresponds to the probability of one myocardial segment.

[0068] In the above scheme, when identifying the target classification of the myocardium, it is possible to identify the probability that the entire myocardium belongs to the target classification, and also to identify the probability that each myocardial segment belongs to the target classification, thereby improving the flexibility and accuracy of target classification identification.

[0069] In some embodiments, Figure 6 As shown, the method of determining the probability that the at least two frames of images to be identified belong to the target category according to the temporal features and the spatial features may include the following steps:

[0070] S131, inputting temporal features and spatial features into a multi-label classification model to determine the probability that the at least two frames of images to be identified belong to the target category.

[0071] Among them, the multi-label classification model is generated by training based on the temporal characteristics of multiple segments of second training set dynamic images, the spatial characteristics of multiple segments of second training set dynamic images, and the classification label of each myocardial segment corresponding to each segment of second training set dynamic images; or, it is generated by training based on the temporal characteristics of multiple segments of second training set dynamic images, the spatial characteristics of multiple segments of second training set dynamic images, and the classification label corresponding to the entire myocardium in each segment of second training set dynamic images; the second training set dynamic images include the myocardial image of the second subject.

[0072] Among them, the classification label is related to the target classification. For example, when the target classification is myocardial ischemia, and the myocardium in the second training set dynamic image belongs to the myocardial ischemic state, the classification label of the second training set dynamic image is determined to be 1, and when the myocardium in the second training set dynamic image does not belong to the myocardial ischemic state, the classification label of the second training set dynamic image is determined to be 0. For another example, when the target classification is myocardial ischemia, and the first segment of the myocardium in the second training set dynamic image belongs to the myocardial ischemic state, and the second segment does not belong to the myocardial ischemic state, the classification label of the first segment in the second training set dynamic image is determined to be 1, and the classification label of the second segment is determined to be 0.

[0073] In some embodiments, Figure 7As shown, the method of training and generating a multi-label classification model can include the following steps:

[0074] S1311 , obtaining multiple segments of second training set dynamic images and a classification label corresponding to each segment of the second training set dynamic image.

[0075] First, the second training set dynamic images are obtained. Specifically, the method of obtaining the second training set dynamic images is the same as the method of obtaining the first training set dynamic images in step S21, which will not be repeated here.

[0076] Secondly, the classification label corresponding to each segment of the second training set dynamic image is obtained. Specifically, the classification label corresponding to each segment of the second training set dynamic image can be a manually marked classification label.

[0077] S1312: Determine the temporal features and spatial features of each segment of the second training set dynamic image.

[0078] Specifically, the method for determining the temporal features and spatial features of each segment of the second training set dynamic image is to input each segment of the second training set dynamic image into the spatiotemporal recognition model trained in steps S21 to S23, thereby obtaining the temporal features and spatial features of each segment of the second training set dynamic image.

[0079] S1313, taking the temporal features and spatial features corresponding to the dynamic images of the second training set as input, and the classification labels corresponding to the dynamic images of the second training set as output targets, and training to generate a multi-label classification model.

[0080] Specifically, the spatial features and temporal features corresponding to the dynamic images of the second training set are used as the Key and Value in the attention mechanism, and the classification label is used as the Query. The attention mechanism is used to perform classification training on the preset model to obtain a multi-label classification model.

[0081] In the above scheme, the temporal features, spatial features and classification labels of dynamic images can be used to train and generate a multi-label classification model that can identify the probability that the dynamic images belong to the target classification, thereby avoiding the problem of manual participation in the process of target classification of dynamic images, saving recognition time and improving the accuracy of the recognition results.

[0082] In some embodiments,

[0083] like Figure 8 As shown, the method of training and generating a multi-label classification model can also include the following steps:

[0084] S1321. Acquire multiple segments of second training set dynamic images and a classification label of each myocardial segment corresponding to each segment of the second training set dynamic image.

[0085] First, the second training set dynamic images are obtained. Specifically, the method of obtaining the second training set dynamic images is the same as the method of obtaining the first training set dynamic images in step S21, which will not be repeated here.

[0086] Secondly, the classification label of each myocardial segment corresponding to each second training set dynamic image is obtained. Specifically, the classification label of each myocardial segment corresponding to each second training set dynamic image can be a manually marked classification label of each myocardial segment.

[0087] S1322. Determine the temporal features and spatial features of each segment of the second training set dynamic image.

[0088] Specifically, the method for determining the temporal features and spatial features of each segment of the second training set dynamic image is to input each segment of the second training set dynamic image into the spatiotemporal recognition model trained in steps S21 to S23, thereby obtaining the temporal features and spatial features of each segment of the second training set dynamic image.

[0089] S1323. Take the temporal features and spatial features corresponding to the dynamic images of the second training set as input, and the classification label of each myocardial segment corresponding to the dynamic images of the second training set as output target, and train to generate a multi-label classification model.

[0090] For example, Fig. 9 As shown, the spatial features and temporal features corresponding to the dynamic images of the second training set are used as the Key and Value in the attention mechanism, the classification label of each myocardial segment is used as the Query, and the preset model is classified and trained using the attention mechanism to obtain a multi-label classification model.

[0091] In the above scheme, the temporal features and spatial features of the dynamic image, as well as the classification labels of each myocardial segment corresponding to the dynamic image, can be used to train and generate a multi-label classification model that can identify the probability of each myocardial segment in the dynamic image belonging to the target classification, thereby avoiding the problem of manual participation in the process of target classification of dynamic images, saving recognition time, and improving the accuracy of the recognition results. That is, when identifying myocardial ischemia, it is no longer just identifying the probability of myocardial ischemia as a whole, but can accurately identify the ischemia probability corresponding to each myocardial segment, thereby improving the recognition accuracy.

[0092] In some embodiments, the preset model may be a vision transformer model (Vision Transformer, VIT).

[0093] The embodiment of the present application can divide the functional modules of the target classification device based on image recognition according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0094] Fig.10 A schematic diagram of the structure of a target classification device based on image recognition provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the target classification device based on image recognition includes an acquisition module 101 , a determination module 102 , and a recognition module 103 .

[0095] The acquisition module 101 is used to acquire at least two frames of images to be identified including the myocardium of the same subject to be detected; the determination module 102 is used to determine the temporal characteristics and spatial characteristics of the images to be identified; the temporal characteristics are used to characterize the time sequence of the at least two frames of images to be identified, and the spatial characteristics are used to characterize the distribution of the myocardium in each frame of the image to be identified; the identification module 103 is used to determine the probability that the at least two frames of images to be identified belong to the target classification based on the temporal characteristics and the spatial characteristics.

[0096] In some embodiments, the determination module 102 is specifically used to input at least two frames of images to be identified into a spatiotemporal recognition model to obtain temporal features and spatial features of the images to be identified; the spatiotemporal recognition model is generated based on training of multiple segments of first training set dynamic images, and the first training set dynamic images include myocardial images of the first subject.

[0097] In some embodiments, Fig.11As shown, the target classification device based on image recognition also includes a processing module 104 and a training module 105; the acquisition module 101 is also used to acquire multiple segments of first training set dynamic images before determining the temporal features and spatial features of the image to be identified; a segment of the first training set dynamic image includes at least two frames of historical images of the myocardium of a first subject; the processing module 104 is used to perform processing operations on the historical images of the first training set dynamic images for each segment of the first training set dynamic image, so as to obtain a disordered image and a missing image corresponding to each segment of the first training set dynamic image; wherein the processing operation includes: performing a time sequence disruption processing on the historical images in the first training set dynamic images to obtain a disordered image; performing a random position masking processing on each frame of the historical image in the first training set dynamic images to obtain a missing image; the training module 105 is used to take the disordered image and the missing image corresponding to the first training set dynamic image as input, and the time sequence and historical images of the historical images in the first training set dynamic images as output targets, and train to obtain a spatiotemporal recognition model.

[0098] In some embodiments, the processing module 104 is further used to pre-process each first training set dynamic image according to preset rules to obtain a historical image corresponding to each first training set dynamic image; the preset rules are used to limit the specifications of the historical images corresponding to the first training set dynamic images.

[0099] In some embodiments, the recognition module 103 is specifically used to determine the probability that the entire myocardium in the at least two frames of images to be recognized belongs to the target category as the probability that the at least two frames of images to be recognized belong to the target category; or, to determine the probability that each myocardial segment in the at least two frames of images to be recognized belongs to the target category as the probability that the at least two frames of images to be recognized belong to the target category.

[0100] In some embodiments, the recognition module 103 is specifically used to input the temporal features and the spatial features into a multi-label classification model to obtain the probability that at least two frames of images to be identified belong to the target category; the multi-label classification model is generated by training based on the temporal features of multiple segments of second training set dynamic images, the spatial features of multiple segments of second training set dynamic images, and the classification labels of each myocardial segment corresponding to each segment of second training set dynamic images; or, the multi-label classification model is generated by training based on the temporal features of multiple segments of second training set dynamic images, the spatial features of multiple segments of second training set dynamic images, and the classification labels corresponding to the entire myocardium in each segment of second training set dynamic images; the second training set dynamic images include the myocardial images of the second subject.

[0101] The target classification device based on image recognition provided in this embodiment can execute the target classification method based on image recognition provided in the above method embodiment. Its implementation principle and technical effect are similar to those of the above method and will not be repeated here.

[0102] Fig.12 An electronic device according to an exemplary embodiment may include a processor 702, and the processor 702 is used to execute application code to implement the target classification method based on image recognition in the present application.

[0103] The processor 702 may be a central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0104] like Fig.12 As shown, the electronic device may further include a memory 703. The memory 703 is used to store application program codes for executing the solution of the present application, and the execution is controlled by the processor 702.

[0105] The memory 703 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 703 may exist independently and be connected to the processor 702 via the bus 704. The memory 703 may also be integrated with the processor 702.

[0106] like Fig.12 As shown, the electronic device may further include a communication interface 701, wherein the communication interface 701, the processor 702, and the memory 703 may be coupled to each other, for example, via a bus 704. The communication interface 701 is used to exchange information with other devices, for example, to support information exchange between the electronic device and other devices.

[0107] It should be pointed out that Fig.12 The device structure shown in the figure does not constitute a limitation on the electronic device, except Fig.12In addition to the components shown, the electronic device may include more or fewer components than shown, or combine certain components, or arrange the components differently. The electronic device provided in this embodiment can execute the target classification method based on image recognition provided in the above method embodiment, and its implementation principle and technical effect are similar to the above method, which will not be repeated here.

[0108] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, each process of the target classification method based on image recognition in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0109] The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0110] An embodiment of the present application provides a computer program product, which stores a computer program. When the computer program is executed by a processor, each process of the target classification method based on image recognition in the above method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0111] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0112] In the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0113] In this application, memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0114] In this application, computer-readable media includes permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. According to the definition in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data and carrier waves.

[0115] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0116] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target classification method based on image recognition, characterized in that: include: Acquire at least two frames of images to be identified including the myocardium of the same subject to be detected; Determining the temporal characteristics and spatial characteristics of the image to be identified; The temporal feature is used to characterize the time sequence of the at least two frames of images to be identified, and the spatial feature is used to characterize the myocardial distribution in each frame of the image to be identified; The probability that the at least two frames of images to be identified belong to the target category is determined according to the temporal features and the spatial features.

2. The target classification method according to claim 1, characterized in that: The determining of the temporal features and spatial features of the image to be identified includes: The at least two frames of images to be identified are input into a spatiotemporal recognition model to obtain temporal features and spatial features of the images to be identified; the spatiotemporal recognition model is generated by training based on multiple segments of first training set dynamic images, wherein the first training set dynamic images include myocardial images of a first subject.

3. The target classification method according to claim 2, characterized in that: Before determining the temporal features and spatial features of the image to be identified, the method further includes: Acquire multiple segments of first training set dynamic images; a segment of the first training set dynamic image includes at least two frames of historical images of the myocardium of a first subject; For each segment of the first training set dynamic image, the historical images of the first training set dynamic image are processed to obtain a disordered image and a missing image corresponding to each segment of the first training set dynamic image; wherein the processing operation includes: performing a time sequence disorder process on the historical images in the first training set dynamic image to obtain a disordered image; performing a random position masking process on each frame of the historical image in the first training set dynamic image to obtain a missing image; The disordered images and the missing images corresponding to the first training set dynamic images are taken as input, and the time sequence of historical images in the first training set dynamic images and the historical images are taken as output targets, and a spatiotemporal recognition model is obtained through training.

4. The target classification method according to claim 3, characterized in that: Before processing the historical images of the first training set dynamic images, the method further includes: Preprocess each first training set dynamic image according to preset rules to obtain a historical image corresponding to each first training set dynamic image; the preset rules are used to limit the specifications of the historical images corresponding to the first training set dynamic images.

5. The target classification method according to claim 1, characterized in that: The determining the probability that the at least two frames of images to be identified belong to the target category includes: Determining the probability that the entire myocardium in the at least two frames of images to be identified belongs to the target category as the probability that the at least two frames of images to be identified belong to the target category; Alternatively, the probability that each myocardial segment in the at least two frames of images to be identified belongs to the target category is determined as the probability that the at least two frames of images to be identified belong to the target category.

6. The target classification method according to claim 5, characterized in that: The determining, based on the temporal features and the spatial features, the probability that the at least two frames of images to be identified belong to the target category includes: Inputting the temporal features and the spatial features into a multi-label classification model to obtain the probability that the at least two frames of images to be identified belong to the target category; The multi-label classification model is generated by training based on the temporal features of multiple segments of second training set dynamic images, the spatial features of multiple segments of second training set dynamic images, and the classification label of each myocardial segment corresponding to each segment of second training set dynamic images; or, the multi-label classification model is generated by training based on the temporal features of multiple segments of second training set dynamic images, the spatial features of multiple segments of second training set dynamic images, and the classification label corresponding to the entire myocardium in each segment of second training set dynamic images; the second training set dynamic images include the myocardial image of the second subject.

7. A target classification device based on image recognition, characterized in that: include: An acquisition module, used for acquiring at least two frames of images to be identified including the myocardium of the same subject to be detected; A determination module, used to determine the temporal characteristics and spatial characteristics of the image to be identified; The temporal feature is used to characterize the time sequence of the at least two frames of images to be identified, and the spatial feature is used to characterize the myocardial distribution in each frame of the image to be identified; The recognition module is used to determine the probability that the at least two frames of images to be recognized belong to the target category according to the temporal characteristics and the spatial characteristics.

8. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the target classification method based on image recognition as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for classifying an object based on image recognition according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that When the computer program product runs on a computer, the computer is enabled to implement the target classification method based on image recognition as claimed in any one of claims 1 to 6.

Images