Image recognition methods, devices, equipment and storage media

Abstract

This application discloses an image recognition method, apparatus, device, and storage medium, belonging to the field of artificial intelligence technology. The method includes: for the nth round of recognition, acquiring the nth image sequence, which includes M two-dimensional medical images, the M two-dimensional medical images being follow-up images obtained by scanning the same body part at N follow-up times; acquiring n-1 prior medical states and n-1 medical progression states obtained from the previous n-1 rounds of recognition; preprocessing and feature extraction of the nth image sequence; and based on the obtained feature data, n-1 prior medical states, and n-1 medical progression states, predicting the nth and n+1th medical progression states of the body part in the nth round of recognition. The image recognition accuracy of this application is high.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image recognition method, apparatus, device, and storage medium. Background Technology

[0002] In the era of Artificial Intelligence (AI), its applications have spread to various fields, including medicine. In the medical field, AI has enabled unprecedented advancements in medical technology, leading to revolutionary changes almost daily.

[0003] In the medical field, machine learning techniques are used to identify medical images and predict the state of a disease. A common technique involves using machine learning to identify individual CT (Computed Tomography) images of lesion areas, such as inputting a single CT image into a convolutional neural network for analysis, thereby predicting the disease's condition.

[0004] The image recognition methods described above rely solely on a single CT image, resulting in limited data and inaccurate predictions. In other words, the accuracy of image recognition is not high. Summary of the Invention

[0005] This application provides an image recognition method, apparatus, device, and storage medium, which can improve the accuracy of image recognition. The technical solution is as follows:

[0006] On the one hand, an image recognition method is provided, the method comprising:

[0007] For the nth round of recognition, the nth image sequence is obtained, which includes M two-dimensional medical images. The M two-dimensional medical images are follow-up images obtained by scanning the same body part at N follow-up times; n and M are integers not less than 2.

[0008] Obtain n-1 preceding medical states and n-1 medical progress states predicted in the previous n-1 rounds of identification; wherein, the preceding medical states are used to indicate the medical condition of the body part in the previous round of identification, and the medical progress states are used to indicate the predicted progress of the medical condition in the next stage in any round of identification.

[0009] The nth image sequence is preprocessed and features are extracted; based on the obtained feature data, the n-1 preceding medical states and the n-1 medical progress states, the nth medical state and the n+1th medical progress state of the body part in the nth round of identification are predicted, and the nth medical state is used to indicate the medical condition of the body part in the nth round of identification.

[0010] On the other hand, an image recognition device is provided, the device comprising:

[0011] The first acquisition module is configured to acquire the nth image sequence for the nth round of recognition process. The nth image sequence includes M two-dimensional medical images, which are follow-up images obtained by scanning the same body part at N follow-up times; n and M are integers not less than 2.

[0012] The second acquisition module is configured to acquire n-1 preceding medical states and n-1 medical progress states predicted in the previous n-1 rounds of identification process; wherein, the preceding medical states are used to indicate the medical condition of the body part in the preceding round of identification process, and the medical progress states are used to indicate the medical condition progress predicted in the next stage in any round of identification process.

[0013] The recognition module is configured to preprocess and extract features from the nth image sequence; based on the obtained feature data, the n-1 preceding medical states and the n-1 medical progress states, predict the nth medical state and the n+1th medical progress state of the body part in the nth round of recognition, wherein the nth medical state is used to indicate the medical condition of the body part in the nth round of recognition.

[0014] In some embodiments, the first acquisition module is configured to determine M follow-up images that are temporally adjacent and most recently scanned, and to use the M follow-up images as the nth image sequence.

[0015] In some embodiments, the identification module is configured to:

[0016] The nth image sequence is preprocessed to obtain a new image;

[0017] The new image is input into the prediction model of the image recognition network for feature extraction, and the feature data output by the prediction model is used as the input to the first fully connected layer of the image recognition network.

[0018] Obtain the first feature vector output by the first fully connected layer; perform feature fusion on the first feature vector, the n-1 preceding medical states, and the n-1 medical progression states, and input the fused second feature vector into the second fully connected layer of the image recognition network;

[0019] After passing through the second fully connected layer, the second feature vector is integrated into the nth medical state and the (n+1)th medical progress state.

[0020] In some embodiments, the n-1 preceding medical states and the n-1 medical progression states are in the form of feature vectors, and the identification module is configured to:

[0021] The first feature vector, the n-1 preceding medical states, and the n-1 medical progress states are concatenated to obtain the second feature vector.

[0022] In some embodiments, the prediction model includes a plurality of sequentially connected residual blocks; wherein each residual block includes an identity mapping and at least two convolutional layers, and the identity mapping of any residual block points from the input of the residual block to the output of the residual block; or,

[0023] The prediction model includes a tight-connected block; wherein, a tight-connected block includes at least two tight-connected layers; for any tight-connected block, the input of each tight-connected layer includes the outputs of all previous tight-connected layers in the tight-connected block.

[0024] In some embodiments, the first convolutional layer, the second convolutional layer, and the third convolutional layer of the at least two convolutional layers are sequentially connected, and the identity mapping points from the input of the first convolutional layer to the output of the third convolutional layer; the recognition module is configured to:

[0025] The new image is input into the first residual block of the prediction model;

[0026] For any residual block, the output of the previous residual block is received, and feature extraction is performed on the output of the previous residual block based on the first convolutional layer, the second convolutional layer, and the third convolutional layer.

[0027] Obtain the output of the third convolutional layer, and pass the output of the third convolutional layer and the output of the previous residual block to the next residual block;

[0028] The feature data is obtained by obtaining the output of the last residual block of the prediction model.

[0029] In some embodiments, the identification module is configured to:

[0030] Image reconstruction is performed on the M two-dimensional medical images to obtain M three-dimensional medical images;

[0031] Acquire slice images of each 3D medical image from different angles;

[0032] Each slice of a 3D medical image is input into a convolutional layer with different channels for feature extraction, resulting in at least two feature maps.

[0033] Data augmentation processing is performed on the at least two feature maps to obtain multiple augmented images;

[0034] The multiple enhanced images are fused to obtain the new image.

[0035] In some embodiments, the first acquisition module is further configured to acquire a first image sequence for the first round of recognition process, the first image sequence including M two-dimensional medical images, the M two-dimensional medical images including a baseline image and M-1 follow-up images;

[0036] The recognition module is further configured to predict, based on the first image sequence, a first medical state and a first medical progress state of the body part during the first round of recognition, wherein the first medical state is used to indicate the medical condition of the body part during the first round of recognition.

[0037] On the other hand, a computer device is provided, the device including a processor and a memory, the memory storing at least one piece of program code, the at least one piece of program code being loaded and executed by the processor to implement the image recognition method described above.

[0038] On the other hand, a computer-readable storage medium is provided, wherein at least one piece of program code is stored in the storage medium, the at least one piece of program code being loaded and executed by a processor to implement the image recognition method described above.

[0039] On the other hand, a computer program product or computer program is provided, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the image recognition method described above.

[0040] For the nth round of recognition, this embodiment of the application inputs an image sequence, n-1 preceding medical states, and n-1 previously obtained medical progression states for prediction. Since multiple two-dimensional medical images included in the image sequence can be processed simultaneously, image information from multiple follow-up images can be accumulated, fully utilizing the information from preceding follow-up images. Furthermore, the prediction also fully utilizes preceding medical states and previously obtained medical progression states, and prediction based on rich input data significantly increases the accuracy of the final prediction result. That is, this embodiment of the application can improve the accuracy of image recognition. In addition, this embodiment of the application can also perform multi-task prediction, such as both determining the current stage of medical condition and predicting the next stage of medical condition progression, achieving collaborative learning of the two tasks. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the implementation environment of an image recognition scheme provided in an embodiment of this application;

[0043] Figure 2 This is an overall architecture diagram of an image recognition scheme provided in an embodiment of this application;

[0044] Figure 3 This is an overall architecture diagram of an image recognition scheme provided in an embodiment of this application;

[0045] Figure 4 This is a flowchart of an image recognition method provided in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram of a dense block structure provided in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram of the structure of a residual block provided in an embodiment of this application;

[0048] Figure 7 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application;

[0049] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0050] Figure 9 This is a schematic diagram of the structure of another computer device provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0053] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms.

[0054] These terms are simply used to distinguish one element from another. For example, without departing from the various examples, the first element can be referred to as the second element, and similarly, the second element can be referred to as the first element. Both the first and second elements can be elements, and in some cases, they can be separate and distinct elements.

[0055] "At least one" means one or more elements. For example, at least one element can be one element, two elements, three elements, or any integer number of elements greater than or equal to one. "At least two" means two or more elements. For example, at least two elements can be two elements, three elements, or any integer number of elements greater than or equal to two.

[0056] The image recognition scheme provided in this application involves artificial intelligence technology.

[0057] Artificial intelligence (AI) encompasses the theories, methods, technologies, and application systems that utilize digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, enabling them to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0058] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0059] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to using cameras and computers to replace human eyes in recognizing and measuring targets, and then performing image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0060] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0061] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0062] The image recognition scheme provided in this application involves technologies such as computer vision and machine learning in artificial intelligence, which will be specifically described through the following embodiments.

[0063] The following describes some key terms or abbreviations that may be involved in the embodiments of this application.

[0064] CT images: Using precisely collimated X-ray beams, gamma rays, or ultrasound, along with highly sensitive detectors, a cross-sectional scan is performed around a specific part of the body to obtain a cross-sectional image. This image can be used to examine various diseases. Depending on the type of radiation used, it can be divided into X-ray CT, gamma-ray CT, etc. Typically, CT scans produce grayscale cross-sectional images, i.e., two-dimensional images. One cross-sectional image represents a section of tissue from the body.

[0065] Isotropy: A physical property can be measured in different directions. If the measurement results are the same in all directions, it means that the physical property is independent of orientation, and this is called isotropy. Conversely, if the physical property is closely related to orientation, and the measurement results are very different in different orientations, this is called anisotropy.

[0066] The implementation environment of the image recognition scheme provided in the embodiments of this application is described below.

[0067] See Figure 1 The implementation environment includes: training equipment 110 and application equipment 120.

[0068] During the training phase, training device 110 is used to train a multi-channel convolutional neural network (also referred to as an image recognition network in text). During the application phase, application device 120 can perform image recognition based on the multi-channel convolutional neural network. For example, the recognition object in this embodiment is a medical image. Furthermore, since image recognition is essentially image classification, this image recognition method can also be called an image classification method.

[0069] Convolutional neural networks (CNNs) are a type of deep learning model. Similar to ordinary neural networks, they consist of neurons with learnable weights and constant biases. They significantly reduce network complexity through local connections and global sharing. Multi-channel CNNs mean that different channels receive different medical image information, which is highly effective for processing medical images that form a temporal sequence.

[0070] Optionally, the training device 110 and application device 120 described above are computer devices, such as terminals or servers. In some embodiments, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, mobile medical device, fixed medical device, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0071] In another embodiment, the training device 110 and the application device 120 may be the same device, or they may be different devices. Furthermore, when the training device 110 and the application device 120 are different devices, they may be of the same type, such as both being terminals; or they may be of different types, such as the training device 110 being a server and the application device 120 being a terminal. This application does not impose any limitations on these aspects.

[0072] The following describes the application scenarios of the image recognition scheme provided in the embodiments of this application.

[0073] The image recognition scheme provided in this application can be used in intelligent healthcare to assist clinical decision-making.

[0074] Example 1: This image recognition scheme can be used to determine or predict the degree of infection of infectious diseases caused by viruses or bacteria.

[0075] Optionally, the image recognition scheme provided in this application embodiment can determine the degree of disease infection in the current stage and predict the degree of disease infection in the next stage based on a multi-channel convolutional neural network.

[0076] In the treatment of patients whose illnesses are caused by viral or bacterial infections, it is necessary to assess the current level of infection and predict the next stage of progression in real time in order to update treatment methods promptly. CT images are particularly important for this assessment. However, in practice, it is difficult to accurately determine the current level of infection and the progression of the disease, and some patients may suddenly become critically ill. Therefore, there is an urgent clinical need for a means to predict the current level of infection and the next stage of progression. Optionally, the image recognition scheme provided in this application embodiment inputs the patient's medical image sequence into a multi-channel convolutional neural network to determine the patient's current level of infection and predict the next stage of progression, thereby assisting clinical decision-making.

[0077] Optionally, the aforementioned medical image sequence may include multiple medical images that are sequentially arranged in time. For example, the aforementioned medical image sequence may be the patient's first few follow-up CT scans, and this application does not impose any limitations on this.

[0078] Example 2: In addition to judging or predicting the degree of infection of infectious diseases, this image recognition scheme can also be used for clinical decision-making for other diseases, such as tumors, which only need to be trained using the corresponding training samples during the training phase.

[0079] In other words, the image recognition scheme provided in this application is a prediction method that can be easily extended to other diseases. For example, this image recognition scheme can also be used based on a multi-channel convolutional neural network to determine the severity of tumor diseases at the current stage and predict the progression of tumor diseases in the next stage.

[0080] It should be noted that the application scenarios described above are for illustrative purposes only and are not intended to limit the scope of this application. In actual implementation, the technical solutions provided in the embodiments of this application can be flexibly applied according to actual needs.

[0081] The image recognition scheme provided in this application will be described below through the following implementation methods.

[0082] First, combine Figure 2 The overall architecture diagram shown illustrates the basic process of the image recognition scheme provided in this application embodiment. Optionally, the medical state is used to represent the disease condition status, i.e., the severity of the disease; the medical progression state is used to represent the degree of disease progression in the next stage. Correspondingly, the preceding medical state can also be called the preceding disease condition status, and the medical progression state can also be called the disease progression state; this application does not impose any limitations here. Taking a medical image sequence input to the image recognition network (i.e., a multi-channel convolutional neural network) that includes two medical images, and the medical images are CT images, as an example, the basic process is as follows:

[0083] 1. The baseline image 0 was obtained from the patient's first CT scan of the affected organ.

[0084] 2. Follow-up images were obtained by performing CT scans on the affected organs after the patient underwent treatment.

[0085] 3. Input the baseline image 0 and the follow-up image 1 into the image recognition network to obtain state 1, which is used to characterize the severity of the disease in the current round of recognition, and predicted state 2, which is used to predict the degree of disease progression in the next stage.

[0086] 4. Follow-up images 2 were obtained by performing CT scans on the affected organs during the patient's follow-up visit after treatment.

[0087] 5. Input follow-up image 1, follow-up image 2, previous disease state (state 1), and predicted state 2 into the image recognition network to obtain state 2, which is used to characterize the severity of the disease in the current round of recognition, and predicted state 3, which is used to predict the degree of disease progression in the next stage.

[0088] 6. Similarly, the images from the first two follow-up visits (follow-up image n-1, follow-up image n), the n-1 preceding disease states (state 1, state 2, ..., state n-1) predicted from the first n-1 rounds of recognition, and the n-1 previously predicted disease progression states (predicted state 2, predicted state 3, ..., predicted state n) are input into the image recognition network to obtain state n, which characterizes the severity of the disease in the current round of recognition, and predicted state n-1, which predicts the degree of disease progression in the next stage.

[0089] based on Figure 2 The architecture diagram shown illustrates that the image recognition scheme provided in this application embodiment can be divided into the following processes: Figure 3 The following four parts are: multi-angle slicing, input enhancement, image fusion, and prediction. Typically, acquired medical images are CT images, i.e., two-dimensional cross-sectional images. If only two-dimensional cross-sectional images are used for disease prediction, some spatial information will be lost, thus affecting the accuracy of image prediction. For multi-angle slicing, for the input two-dimensional medical image (such as a CT image), it is first reconstructed into an isotropic (e.g., isotropic resolution) three-dimensional medical image; then, the three-dimensional medical image is rotated at any angle in three-dimensional space, and several cross-sectional slices from different orientations are extracted from the rotated three-dimensional medical image to obtain slice images to be input into the convolutional layers of multiple channels, thereby effectively acquiring more spatial information. For input enhancement, the obtained slice images undergo data enhancement after passing through the convolutional layers of the image recognition network. For image fusion, this part is used to perform channel fusion on each image obtained after multi-angle slicing and input enhancement processing. Finally, the prediction part determines the current medical condition and predicts the next stage of medical condition progression based on the fused image, the previous medical state, and the medical progress status.

[0090] Figure 4 This is a flowchart of an image recognition method provided in an embodiment of this application. See also... Figure 4 In the application phase, for the nth round of identification, the method flow provided in this application embodiment includes the following steps.

[0091] 401. Obtain the nth image sequence; wherein the nth image sequence includes M two-dimensional medical images, and the M two-dimensional medical images are follow-up images obtained by scanning the same body part at N follow-up times.

[0092] Where n and M are integers not less than 2; optionally, a two-dimensional medical image refers to a medical image from the same part of the body (such as a diseased organ in the human body), such as a two-dimensional CT image. The diseased organ includes lesions, which typically refer to the part of the body where a disease has occurred. For example, if a lobe of the lung is destroyed by tuberculosis bacteria, then this part is a pulmonary tuberculosis lesion.

[0093] The first point that needs to be explained is that, for Figure 2 In each round of recognition shown, this embodiment of the application utilizes a multi-channel convolutional neural network, which can simultaneously process multiple two-dimensional medical images included in the image sequence. This allows for the accumulation of image information from multiple follow-up images, enabling full utilization of information from previous follow-up images and thus increasing the accuracy of the final prediction.

[0094] In some embodiments, for the nth round of identification, obtaining the nth image sequence includes: determining M follow-up images that are temporally adjacent and most recently scanned, and using these M follow-up images as the nth image sequence. The most recently obtained two-dimensional medical image is the most recently scanned two-dimensional medical image. That is, the most recently scanned two-dimensional medical image is used in the current round of identification (the nth round of identification) to ensure the accuracy of the final prediction and obtain a prediction result that better reflects the patient's current actual condition. For example, assuming the most recently scanned M two-dimensional medical images are follow-up image n-1 and follow-up image n, then in the current round of identification, follow-up image n-1 and follow-up image n are used for prediction.

[0095] The second point to clarify is that temporal adjacency refers to adjacency in scan time. For example, such as... Figure 2As shown, baseline image 0 and follow-up image 1 are temporally adjacent; follow-up image 1 and follow-up image 2 are also temporally adjacent; follow-up image n-1 and follow-up image n are also temporally adjacent. Furthermore, these M two-dimensional medical images were obtained by scanning the same lesion region of the same organism at N follow-up times. For example, the M two-dimensional medical images included in this image sequence were obtained by scanning the lung region of the same patient at N follow-up times. Taking N as 2 and the time interval between two follow-ups as one week, one of the two-dimensional medical images included in this image sequence might have been obtained by scanning the patient's lung region on March 1st; another two-dimensional medical image might have been obtained by scanning the patient's lung region on March 8th.

[0096] 402. Obtain n-1 preceding medical states and n-1 medical progress states predicted in the previous n-1 rounds of identification process; wherein, the preceding medical state is used to indicate the medical condition of the body part in the previous round of identification process, and the medical progress state is used to indicate the medical condition progress predicted in the next stage in any round of identification process.

[0097] In some embodiments, the preceding medical status is also referred to as the preceding disease status, which indicates the severity of the disease in the body part during a preceding round of identification, and the medical progress status is also referred to as the disease progress status, which indicates the predicted degree of disease progression in the next stage during any round of identification.

[0098] In the embodiments of this application, the first round of identification process predicts medical state 1 and medical progress state 2; the second round of identification process predicts medical state 2 and medical progress state 3; the third round of identification process predicts medical state 3 and medical progress state 4; and so on, the (n-1)th round of identification process predicts medical state n-1 and medical progress state n; the nth round of identification process predicts medical state n and medical progress state n+1.

[0099] For the nth round of identification process, the n-1 preceding medical states include: medical state 1, medical state 2, medical state 3, ..., medical state n-1; the n-1 medical progress states include: medical progress state 2, medical progress state 3, medical progress state 4, ..., medical progress state n.

[0100] Specifically, for the first round of identification, the first image sequence is obtained, which also includes M two-dimensional medical images, specifically one baseline image and M-1 follow-up images. See, for example... Figure 2The first image sequence includes a baseline image 0 and a follow-up image 1. Then, based on the first image sequence, the first medical state of the body part during the first round of identification and the second medical progression state for the next stage are predicted. The first medical state indicates the severity of the disease in the affected organ during the first round of identification.

[0101] It should be noted that, apart from the first round of identification, the procedures for each subsequent round of identification are similar and can all be implemented by referring to steps 401 to 403. Furthermore, the image recognition scheme provided in this application embodiment can perform multi-task prediction. For example, this application embodiment can predict both the current medical condition of the diseased organ and its progression in the next stage. Optionally, taking infectious diseases as an example, it can determine the patient's current level of infection and predict the patient's progression in the next stage. That is, the application embodiment can simultaneously output the patient's current level of infection and predict the patient's progression in the next stage.

[0102] 403. Perform preprocessing and feature extraction on the nth image sequence; based on the obtained feature data, n-1 preceding medical states and n-1 medical progress states, predict the nth medical state and n+1 medical progress state of the body part in the nth round of recognition; wherein, the nth medical state is used to indicate the medical condition of the body part in the nth round of recognition.

[0103] In some embodiments, see Figure 3 The nth image sequence is preprocessed and features are extracted. Based on the obtained feature data, n-1 preceding medical states and n-1 medical progression states, the nth medical state and n+1 medical progression state of the body part in the nth round of recognition are predicted, including but not limited to the following steps.

[0104] 4031. Preprocess the nth image sequence to obtain a new image.

[0105] Optionally, the preprocessing process includes, but is not limited to, Figure 3 The multi-angle slicing, image enhancement, and image fusion are shown.

[0106] 4031-1. Reconstruct M two-dimensional medical images to obtain M three-dimensional medical images; for any three-dimensional medical image among the M two-dimensional medical images, slice the three-dimensional medical image at least twice from different angles to obtain sliced ​​images of the three-dimensional medical image.

[0107] In this embodiment of the application, image reconstruction refers to reconstructing a two-dimensional medical image into a three-dimensional medical image. Specifically, a two-dimensional medical image is reconstructed to obtain a three-dimensional medical image.

[0108] The reason for the need for image reconstruction is that current medical imaging technology typically obtains two-dimensional medical images directly. For example, doctors observe projection images obtained from different imaging angles or a set of tomographic images to perform qualitative analysis and identify lesions or assess tissue deformities. Since the human anatomy is three-dimensional, doctors need strong image interpretation skills to make accurate diagnoses from two-dimensional images, making the diagnostic results dependent on the doctor's experience. Accurately determining the spatial location, size, geometry, and spatial relationship of lesions or deformities with surrounding tissue structures is difficult relying solely on the doctor's subjective analysis and judgment. Therefore, current clinical diagnosis urgently needs to display medical data directly to doctors in the form of three-dimensional images. In other words, reconstructing two-dimensional medical images into three-dimensional medical images allows for the intuitive display of complex three-dimensional organ structures, which inevitably helps doctors conduct comprehensive and accurate analysis, thereby improving the level of medical diagnosis.

[0109] In some embodiments, image reconstruction is performed on M two-dimensional medical images to obtain M three-dimensional medical images, including but not limited to: for any two-dimensional medical image, image interpolation processing is performed on the two-dimensional medical image to obtain a three-dimensional medical image with the same resolution in all directions; wherein, the same resolution in all directions means that the resolution of the three-dimensional medical image is the same in all directions.

[0110] Optionally, image interpolation methods include, but are not limited to, gray-based interpolation methods, shape-based interpolation methods, and wavelet-based interpolation methods. Gray-based interpolation methods include, but are not limited to, nearest neighbor interpolation, linear interpolation, Lagrange interpolation, spline interpolation, and matching interpolation. For example, this application uses linear interpolation, such as quadratic linear interpolation, for image reconstruction, but this application does not impose limitations. Linear interpolation refers to a method of determining the value of an unknown quantity between two known quantities using a straight line connecting them. In other words, linear interpolation refers to an interpolation method where the interpolation function is a first-order polynomial. Mathematically, quadratic linear interpolation is an extension of linear interpolation of two variable functions on a square grid; its main approach is to first perform linear interpolation in one direction and then in the other direction.

[0111] The first point to clarify is that any three-dimensional medical image, after being sliced ​​from multiple angles, will yield at least two slice images; in other words, at least two slice images can be obtained from a single three-dimensional medical image. In some embodiments, the three-dimensional medical image is sliced ​​at least twice at different angles to obtain slice images of the three-dimensional medical image, including but not limited to: rotating the three-dimensional medical image at any angle in three-dimensional space; and extracting at least two cross-sectional slices from different orientations from the rotated three-dimensional medical image to obtain slice images of the three-dimensional medical image.

[0112] Optionally, the number of slice images in each of the M three-dimensional medical images is the same, and this application does not impose any restrictions on this. For example, the number of slice images in each three-dimensional medical image is x, where x is a positive integer not less than 2; correspondingly, the total number of slice images obtained is N*x.

[0113] 4031-2. Input the slice images of each three-dimensional medical image into the convolutional layers of different channels of the image recognition network for feature extraction to obtain multiple feature maps.

[0114] The number of feature maps obtained is the same as the number of slice images; that is, a slice image is processed by a convolutional layer to extract features, resulting in one feature map.

[0115] It should be noted that the image recognition network includes convolutional layers with multiple channels. Optionally, the number of channels in the convolutional layer is the same as the number of images included in the image sequence, i.e., the number of channels in the convolutional layer is N. For example, as shown... Figure 3 As shown, the image sequence includes two two-dimensional medical images, namely follow-up image n-1 and follow-up image n. Therefore, the number of convolutional layer channels in this image recognition network is 2.

[0116] Furthermore, for each channel, the convolutional layer of that channel extracts features from each slice of the same 3D medical image. For example, assuming the follow-up image n-1 corresponds to three slice images, then all three slice images will be processed... Figure 3 The first convolutional layer in the algorithm extracts three feature maps.

[0117] In addition, such as Figure 3 As shown, after feature extraction through different convolutional layers to obtain feature maps for different channels, this embodiment of the application will also perform data augmentation processing on the feature maps for different channels.

[0118] 4031-3. Perform data augmentation on at least two feature maps to obtain multiple augmented images.

[0119] In some embodiments, data augmentation processing is performed on at least two feature maps, including: for each of the at least two feature maps, performing at least one of rotation, mirroring, scaling, cropping, and shifting processing on the feature map. Optionally, the rotation processing can be random rotation. It should be noted that in this application embodiment, the feature map after data augmentation processing is referred to as an augmented image. Moreover, after data augmentation processing, a single feature map may yield multiple corresponding augmented images. Additionally, as... Figure 3 As shown, after data augmentation processing, the process proceeds to the multi-channel fusion step.

[0120] 4031-4. Perform image fusion processing on multiple enhanced images to obtain a new image.

[0121] In some embodiments, multi-channel fusion processing is performed on multiple enhanced images, including but not limited to: feature stitching processing of multiple enhanced images. Optionally, embodiments of this application perform feature stitching processing on multiple enhanced images along the channel dimension. For example, taking feature maps F1 and F2 as examples, assuming that the dimension of feature map F1 is (x1, y1, z1) and the dimension of feature map F2 is (x2, y2, z2), where x1 = x2 and y1 = y2, z represents the channel dimension, and x and y represent the width and height of the image, then the dimension of the new feature map F obtained by feature stitching feature maps F1 and F2 is (x1, y1, z1 + z2), and the new feature map F is the new image input to the prediction model.

[0122] In addition, the embodiments of this application can also perform multi-task prediction, such as being able to determine the degree of disease infection at the current stage and predict the degree of disease progression in the next stage, thus achieving collaborative learning of the two tasks.

[0123] 4032. Input the new image into the prediction model of the image recognition network for feature extraction, and use the feature data output by the prediction model as the input of the first fully connected layer of the image recognition network; obtain the first feature vector output by the first fully connected layer; perform feature fusion on the first feature vector, n-1 preceding medical states and n-1 medical progression states, and input the fused second feature vector into the second fully connected layer of the image recognition network; after passing through the second fully connected layer, integrate the second feature vector into the nth medical state and the n+1th medical progression state.

[0124] The structures of the first fully connected layer and the second fully connected layer can be the same or different; this application does not impose any restrictions on this. See also Figure 3, after the feature data output by the prediction model passes through the first fully connected layer, n - 1 previous medical states and n - 1 medical progress states will be added for feature fusion. The fused feature vector will then pass through the second fully connected layer to finally obtain the current medical state n and the medical progress state n + 1 of the next stage. Optionally, after the feature data output by the prediction model passes through the first fully connected layer, a one-dimensional feature vector will be obtained.

[0125] Optionally, the above feature fusion operation is vector concatenation, that is, the first feature vector, n - 1 previous medical states and n - 1 medical progress states are concatenated to obtain a second feature vector. In some embodiments, to ensure that the length of each vector concatenation is the same, fixed-length encoding can be performed on the previous medical states and the previously predicted medical progress states. Assume the fixed length is m (usually 20), and the feature vectors R and P are used to represent the previous medical state and the previously obtained medical progress state respectively. For the nth round of recognition process (n < m), the first n values of the feature vectors R and the feature vector P are represented by consecutive numbers between 0 and 1 (for example, the closer the value is to 1, the more serious the condition), and the subsequent m - n values are all -1.

[0126] Among them, the prediction model in the embodiments of this application is responsible for predicting the progress of the medical condition in the next stage and outputs feature data to the first fully connected layer. The first fully connected layer and the second fully connected layer play the role of mapping the distributed feature representation to the sample label space, aiming to reduce the influence of feature positions on classification. In actual use, the first fully connected layer and the second fully connected layer can be implemented by convolution operations, which are not limited in this application.

[0127] In some embodiments, the prediction model is composed of a convolutional neural network based on a dense block (Dense Block). Among them, when training a relatively deep or multi-layer convolutional neural network, a degradation problem usually occurs, that is, the training effect of the deep network is worse than that of the shallow network. And Dense Block is a deep neural network that stacks features of different layers to improve the training convergence ability and reduce parameters, and can effectively reduce the vanishing gradient and strengthen the feature transmission. Optionally, see Figure 5 , the prediction model includes: a feature extraction block, at least two dense connection blocks (Dense Block), and a transition block (Transition Block) located between any two dense connection blocks; among them, the feature extraction block includes a convolutional layer for feature extraction; the transition block includes a convolutional layer and a pooling layer, for example, a transition block includes a convolutional layer and a pooling layer; each dense connection block includes at least two dense connection layers (Dense Layer). Among them, as Figure 5As shown, each DenseLayer is a dense connection, meaning that the input of each layer is the concatenation of the outputs of all preceding layers. In other words, for any densely connected block, the input of each densely connected layer in that block includes the outputs of all preceding densely connected layers.

[0128] In other embodiments, the prediction model is constructed from a residual block-based convolutional neural network. When the number of network layers reaches a certain point, the network performance saturates, and further increases in depth lead to performance degradation. However, this degradation is not caused by overfitting, as both training and testing accuracy decrease, indicating that neural networks become difficult to train beyond a certain depth. ResNet was developed to address this performance degradation problem as network depth increases. ResNet proposes a Deep Residual Learning (DRL) framework to solve this performance degradation issue caused by increasing network depth.

[0129] Optionally, each residual block includes an identity mapping layer and at least two convolutional layers. The identity mapping of any residual block points from the input to the output of any residual block. Specifically, for any set A, if the mapping f: A → A is defined as f(a) = a, meaning that each element a in A corresponds to itself, then f is called the identity mapping on A.

[0130] Suppose the input to a neural network is x, and the desired network layer relationship mapping is H(x). Let stacked nonlinear layers fit another mapping F(x) = H(x) - x. Then the original mapping H(x) becomes F(x) + x. Assuming that optimizing the residual mapping F(x) is easier than optimizing the original mapping H(x), we first calculate the residual mapping F(x). Then the original mapping is F(x) + x, and F(x) + x can be implemented using shortcut connections.

[0131] Figure 6 A schematic diagram of a residual block is shown. Figure 6 As shown, each residual block includes an identity mapping and at least two convolutional layers. The identity mapping of a residual block points from its input to its output.

[0132] In some embodiments, the first, second, and third convolutional layers in at least two convolutional layers are sequentially connected, and the identity mapping points from the input of the first convolutional layer to the output of the third convolutional layer. Inputting a new image into the prediction model of the image recognition network for feature extraction includes: inputting the new image into the first residual block of the prediction model; for any residual block, receiving the output of the previous residual block and extracting features from the output of the previous residual block based on the first, second, and third convolutional layers; obtaining the output of the third convolutional layer and passing the output of the third convolutional layer and the output of the previous residual block to the next residual block; obtaining the output of the last residual block of the prediction model to obtain feature data.

[0133] That is, by adding an identity mapping, the original function H(x) to be learned is transformed into F(x) + x. Although these two expressions achieve the same effect, the optimization difficulty is different. By using reformulation to decompose a problem into a residual problem between multiple scales, the optimization training effect can be effectively achieved. Figure 6 As shown, this residual block is implemented through Shortcut connections. By superimposing the input and output of this residual block through Shortcut connections, the training speed of the model is greatly increased and the training effect is improved without adding extra parameters and computation to the network. Furthermore, this simple structure can effectively solve the degradation problem when the number of layers in the model increases.

[0134] In other words, H(x) is a complex latent mapping of expectation, which is difficult to learn. If we directly... Figure 6 The Shortcut connection in ResNet passes the input x to the output as the initial result. Therefore, the target to be learned is F(x) = H(x) - x. Thus, the ResNet network essentially changes its learning objective; it no longer learns a complete output, but rather the difference between the optimal solution H(x) and the identity mapping x, i.e., the residual mapping F(x). It's important to note that "Shortcut" originally refers to a shortcut, but in this paper it refers to a connection that bypasses layers. In ResNet, the Shortcut connection has no weights; after passing x, each residual block only learns the residual mapping F(x). Furthermore, because the network is stable and easy to learn, its performance gradually improves with increasing network depth. Therefore, when the network has sufficient layers, optimizing the residual mapping F(x) = H(x) - x will easily optimize a complex nonlinear mapping H(x).

[0135] Based on the above description, it can be seen that, compared to ordinary directly connected convolutional neural networks, ResNet networks have many bypass branches that directly connect the input to subsequent layers, allowing those layers to directly learn the residuals. This structure is called Shortcut Connection. Traditional convolutional or fully connected layers often suffer from information loss or degradation during information transmission. ResNet networks solve this problem to some extent by directly routing the input to the output, preserving the integrity of the information. The entire network only needs to learn the difference between the input and output, simplifying the learning objective and reducing the difficulty.

[0136] The image recognition scheme provided in this application has at least the following beneficial effects:

[0137] Multi-angle slicing involves taking multiple slices from different angles within a 3D medical image and inputting all the resulting sliced ​​images into an image recognition network for prediction. This approach utilizes almost all the spatial information of the 3D image, suppressing overfitting and resulting in more accurate predictions, thus achieving higher image recognition accuracy and better performance. Furthermore, multi-angle slicing effectively enhances the input data, suppressing overfitting and simplifying the design and training of the image recognition network. Additionally, employing a multi-channel convolutional neural network allows for the simultaneous processing of multiple 2D medical images within a sequence, accumulating image information from multiple follow-up images and fully utilizing information from previous follow-up images, further increasing the accuracy of the prediction results. Moreover, the prediction process also fully leverages previous medical states and previously obtained medical progression status, further enhancing the accuracy of the prediction results. Furthermore, it enables multi-task prediction, such as determining the current stage of disease infection severity and predicting the next stage of disease progression, achieving collaborative learning between the two tasks.

[0138] In other embodiments, in order to improve the accuracy of prediction, steps 401 to 403 above may be repeated several times in the current round of identification.

[0139] For example, repeating steps 401 to 403 above 10 times yields 10 judgment results of the current medical status and 10 prediction results of the next stage of medical condition progression. The final judgment result and the final prediction result are obtained by averaging these results, meaning each result has the same weighting value. Alternatively, if prior knowledge exists, a weighted average can be used to obtain the final judgment result and the final prediction result, where the weighting value for each result is different; this application does not impose such limitations. That is, the nth round of identification includes at least two identifications based on the nth image sequence. This application embodiment further includes: obtaining at least two identification results from the nth round of identification; wherein each identification result includes a judgment result and a prediction result; the judgment result indicates the current medical status of the diseased organ, and the prediction result indicates the next stage of medical condition progression of the diseased organ; performing a weighted average on the at least two judgment results, and predicting the current medical status of the diseased organ based on the obtained first mean; performing a weighted average on the at least two prediction results, and predicting the next stage of medical condition progression of the diseased organ based on the obtained second mean.

[0140] In other embodiments, during the training phase, the training process of the above-mentioned image recognition network (also known as a multi-channel convolutional neural network) includes, but is not limited to, the following steps.

[0141] A training sample set is obtained, comprising multiple sample image sequences. Each sample image sequence includes multiple temporally adjacent two-dimensional medical sample images obtained by scanning the same body part at multiple different time points. These multiple sample image sequences are then input into an initial network in batches for medical status prediction. The prediction results output by the initial network corresponding to each sample image sequence are obtained. These prediction results include the severity of the condition at the current stage and the progression of the medical condition in the next stage. Based on the prediction results of the sample image sequences and the corresponding standard medical status, a loss function is constructed. In response to the loss value of the loss function not meeting a preset condition, the network parameters of the initial network are iteratively updated repeatedly until the preset condition is met, thus obtaining the image recognition network. Optionally, the preset condition is that the loss value (also called the error value) is less than a set threshold. This set threshold can be set according to actual needs, such as based on model accuracy; this application does not impose any restrictions on this.

[0142] Figure 7 This is a schematic diagram of the structure of an image recognition device provided in an embodiment of this application. See also... Figure 7 The device includes:

[0143] The first acquisition module 701 is configured to acquire the nth image sequence for the nth round of recognition process. The nth image sequence includes M two-dimensional medical images, which are follow-up images obtained by scanning the same body part at N follow-up times; n and M are integers not less than 2.

[0144] The second acquisition module 702 is configured to acquire n-1 preceding medical states and n-1 medical progress states predicted in the previous n-1 rounds of identification process; wherein, the preceding medical states are used to indicate the medical condition of the body part in the preceding round of identification process, and the medical progress states are used to indicate the medical condition progress predicted in the next stage in any round of identification process.

[0145] The recognition module 703 is configured to preprocess and extract features from the nth image sequence; based on the obtained feature data, the n-1 preceding medical states and the n-1 medical progress states, predict the nth medical state and the n+1th medical progress state of the body part in the nth round of recognition, wherein the nth medical state is used to indicate the medical condition of the body part in the nth round of recognition.

[0146] For the nth round of recognition, this embodiment of the application inputs an image sequence, n-1 preceding medical states, and n-1 previously obtained medical progression states for prediction. Since multiple two-dimensional medical images included in the image sequence can be processed simultaneously, image information from multiple follow-up images can be accumulated, fully utilizing the information from preceding follow-up images. Furthermore, the prediction also fully utilizes preceding medical states and previously obtained medical progression states, and prediction based on rich input data significantly increases the accuracy of the final prediction result. That is, this embodiment of the application can improve the accuracy of image recognition. In addition, this embodiment of the application can also perform multi-task prediction, such as both determining the current stage of medical condition and predicting the next stage of medical condition progression, achieving collaborative learning of the two tasks.

[0147] In some embodiments, the first acquisition module is configured to determine M follow-up images that are temporally adjacent and most recently scanned, and to use the M follow-up images as the nth image sequence.

[0148] In some embodiments, the identification module is configured to:

[0149] The nth image sequence is preprocessed to obtain a new image;

[0150] The new image is input into the prediction model of the image recognition network for feature extraction, and the feature data output by the prediction model is used as the input to the first fully connected layer of the image recognition network.

[0151] Obtain the first feature vector output by the first fully connected layer; perform feature fusion on the first feature vector, the n-1 preceding medical states, and the n-1 medical progression states, and input the fused second feature vector into the second fully connected layer of the image recognition network;

[0152] After passing through the second fully connected layer, the second feature vector is integrated into the nth medical state and the (n+1)th medical progress state.

[0153] In some embodiments, the n-1 preceding medical states and the n-1 medical progression states are in the form of feature vectors, and the identification module is configured to:

[0154] The first feature vector, the n-1 preceding medical states, and the n-1 medical progress states are concatenated to obtain the second feature vector.

[0155] In some embodiments, the prediction model includes a plurality of sequentially connected residual blocks; wherein each residual block includes an identity mapping and at least two convolutional layers, and the identity mapping of any residual block points from the input of the residual block to the output of the residual block; or,

[0156] The prediction model includes a tight-connected block; wherein, a tight-connected block includes at least two tight-connected layers; for any tight-connected block, the input of each tight-connected layer includes the outputs of all previous tight-connected layers in the tight-connected block.

[0157] In some embodiments, the first convolutional layer, the second convolutional layer, and the third convolutional layer of the at least two convolutional layers are sequentially connected, and the identity mapping points from the input of the first convolutional layer to the output of the third convolutional layer; the recognition module is configured to:

[0158] The new image is input into the first residual block of the prediction model;

[0159] For any residual block, the output of the previous residual block is received, and feature extraction is performed on the output of the previous residual block based on the first convolutional layer, the second convolutional layer, and the third convolutional layer.

[0160] Obtain the output of the third convolutional layer, and pass the output of the third convolutional layer and the output of the previous residual block to the next residual block;

[0161] The feature data is obtained by obtaining the output of the last residual block of the prediction model.

[0162] In some embodiments, the identification module is configured to:

[0163] Image reconstruction is performed on the M two-dimensional medical images to obtain M three-dimensional medical images;

[0164] Acquire slice images of each 3D medical image from different angles;

[0165] Each slice of a 3D medical image is input into a convolutional layer with different channels for feature extraction, resulting in at least two feature maps.

[0166] Data augmentation processing is performed on the at least two feature maps to obtain multiple augmented images;

[0167] The multiple enhanced images are fused to obtain the new image.

[0168] In some embodiments, the first acquisition module is further configured to acquire a first image sequence for the first round of recognition process, the first image sequence including M two-dimensional medical images, the M two-dimensional medical images including a baseline image and M-1 follow-up images;

[0169] The recognition module is further configured to predict, based on the first image sequence, a first medical state and a first medical progress state of the body part during the first round of recognition, wherein the first medical state is used to indicate the medical condition of the body part during the first round of recognition.

[0170] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0171] It should be noted that the image recognition device provided in the above embodiments is only illustrated by the division of the above functional modules when recognizing images. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image recognition device and the image recognition method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0172] Figure 8 This illustration shows a structural block diagram of a computer device 800 provided in an exemplary embodiment of this application. Taking a computer device as an example, the computer device 800 includes a processor 801 and a memory 802.

[0173] Processor 801 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0174] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 802 are used to store at least one program code, which is executed by the processor 801 to implement the image recognition method provided in the method embodiments of this application.

[0175] In some embodiments, the computer device 800 may also optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 803 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, and a power supply 809.

[0176] Peripheral device interface 803 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 801 and memory 802. In some embodiments, processor 801, memory 802 and peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 801, memory 802 and peripheral device interface 803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0177] The radio frequency (RF) circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 804 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 804 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 804 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0178] Display screen 805 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 805 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 801 for processing. In this case, display screen 805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 805, disposed on the front panel of computer device 800; in other embodiments, there may be at least two display screens, disposed on different surfaces of computer device 800 or in a folded design; in still other embodiments, display screen 805 may be a flexible display screen, disposed on a curved or folded surface of computer device 800. Furthermore, display screen 805 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 805 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0179] The camera assembly 806 is used to acquire images or videos. Optionally, the camera assembly 806 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0180] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 801 for processing, or input to the radio frequency circuit 804 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located in a different part of the computer device 800. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0181] Power supply 809 is used to supply power to various components in computer device 800. Power supply 809 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 809 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0182] In some embodiments, the computer device 800 further includes one or more sensors 810. The one or more sensors 810 include, but are not limited to, an accelerometer 811, a gyroscope 812, a pressure sensor 813, an optical sensor 815, and a proximity sensor 816.

[0183] Accelerometer 811 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by computer device 800. For example, accelerometer 811 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 801 can control display screen 805 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 811. Accelerometer 811 can also be used for games or for acquiring user motion data.

[0184] The gyroscope sensor 812 can detect the orientation and rotation angle of the computer device 800. The gyroscope sensor 812, in conjunction with the accelerometer sensor 811, can collect 3D motion data from the user on the computer device 800. Based on the data collected by the gyroscope sensor 812, the processor 801 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0185] The pressure sensor 813 can be disposed on the side bezel of the computer device 800 and / or on the lower layer of the display screen 805. When the pressure sensor 813 is disposed on the side bezel of the computer device 800, it can detect the user's grip signal on the computer device 800, and the processor 801 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 813. When the pressure sensor 813 is disposed on the lower layer of the display screen 805, the processor 801 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 805. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0186] An optical sensor 815 is used to collect ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 based on the ambient light intensity collected by the optical sensor 815. Specifically, when the ambient light intensity is high, the display brightness of the display screen 805 is increased; when the ambient light intensity is low, the display brightness of the display screen 805 is decreased. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 based on the ambient light intensity collected by the optical sensor 815.

[0187] A proximity sensor 816, also known as a distance sensor, is typically mounted on the front panel of a computer device 800. The proximity sensor 816 is used to detect the distance between the user and the front of the computer device 800. In one embodiment, when the proximity sensor 816 detects that the distance between the user and the front of the computer device 800 is gradually decreasing, the processor 801 controls the display screen 805 to switch from a screen-on state to a screen-off state; when the proximity sensor 816 detects that the distance between the user and the front of the computer device 800 is gradually increasing, the processor 801 controls the display screen 805 to switch from a screen-off state to a screen-on state.

[0188] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the computer device 800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0189] Figure 9This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Taking a computer device as a server as an example, the server 900 can vary considerably due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 901 and one or more memories 902, wherein the memory 902 stores at least one piece of program code, which is loaded and executed by the processor 901 to implement the image recognition method provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0190] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including program code that can be executed by a processor in a terminal to perform the image recognition method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, or optical data storage device, etc.

[0191] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer program code stored in a computer-readable storage medium. The processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the image recognition method described above.

[0192] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0193] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image recognition method, characterized in that, The method includes: For the nth round of recognition, the nth image sequence is obtained, which includes M two-dimensional medical images. The M two-dimensional medical images are follow-up images obtained by scanning the same body part at N follow-up times; n and M are integers not less than 2. Obtain n-1 preceding disease states and n-1 disease progression states predicted in the previous n-1 rounds of identification; wherein, the preceding disease states are used to indicate the disease condition of the body part in the previous round of identification, and the disease progression states are used to indicate the disease progression predicted in the next stage in any round of identification. The nth image sequence is preprocessed to obtain a new image; the preprocessing process includes multi-angle slicing, image enhancement, and image fusion. The new image is input into the prediction model of the image recognition network for feature extraction, and the feature data output by the prediction model is used as the input to the first fully connected layer of the image recognition network. Obtain the first feature vector output by the first fully connected layer; when the n-1 preceding disease states and the n-1 disease progression states are in the form of feature vectors, concatenate the first feature vector, the n-1 preceding disease states, and the n-1 disease progression states to obtain the second feature vector; The second feature vector is input into the second fully connected layer of the image recognition network; after passing through the second fully connected layer, the second feature vector is integrated into the nth disease state and the (n+1)th disease progression state of the body part in the nth round of recognition, wherein the nth disease state is used to indicate the disease condition of the body part in the nth round of recognition.

2. The method according to claim 1, characterized in that, The process of obtaining the nth image sequence includes: The M follow-up images that are temporally adjacent and most recently scanned are determined, and these M follow-up images are used as the nth image sequence.

3. The method according to claim 1, characterized in that, The prediction model includes multiple sequentially connected residual blocks; wherein each residual block includes an identity mapping and at least two convolutional layers, and the identity mapping of any residual block points from the input to the output of that residual block; or, The prediction model includes a tight-connected block; wherein, a tight-connected block includes at least two tight-connected layers; for any tight-connected block, the input of each tight-connected layer includes the outputs of all previous tight-connected layers in the tight-connected block.

4. The method according to claim 3, characterized in that, The first, second, and third convolutional layers in the at least two convolutional layers are connected sequentially, and the identity mapping points from the input of the first convolutional layer to the output of the third convolutional layer; The step of inputting the new image into the prediction model of the image recognition network for feature extraction includes: The new image is input into the first residual block of the prediction model; For any residual block, the output of the previous residual block is received, and feature extraction is performed on the output of the previous residual block based on the first convolutional layer, the second convolutional layer, and the third convolutional layer. Obtain the output of the third convolutional layer, and pass the output of the third convolutional layer and the output of the previous residual block to the next residual block; The feature data is obtained by obtaining the output of the last residual block of the prediction model.

5. The method according to claim 1, characterized in that, The preprocessing of the nth image sequence to obtain a new image includes: Image reconstruction is performed on the M two-dimensional medical images to obtain M three-dimensional medical images; Acquire slice images of each 3D medical image from different angles; Each slice of a 3D medical image is input into a convolutional layer with different channels for feature extraction, resulting in at least two feature maps. Data augmentation processing is performed on the at least two feature maps to obtain multiple augmented images; The multiple enhanced images are fused to obtain the new image.

6. The method according to any one of claims 1 to 5, wherein the method further comprises: For the first round of identification, the first image sequence is obtained, which includes M two-dimensional medical images that are different from the nth image sequence. The first image sequence includes a baseline image and M-1 follow-up images. Based on the first image sequence, the first disease state of the body part in the first round of identification and the disease progression state in the next stage are predicted, wherein the first disease state is used to indicate the disease state of the body part in the first round of identification.

7. An image recognition device, characterized in that, The device includes: The first acquisition module is configured to acquire the nth image sequence for the nth round of recognition process. The nth image sequence includes M two-dimensional medical images, which are follow-up images obtained by scanning the same body part at N follow-up times; n and M are integers not less than 2. The second acquisition module is configured to acquire n-1 preceding disease states and n-1 disease progression states predicted in the previous n-1 rounds of identification; wherein, the preceding disease states are used to indicate the disease condition of the body part in the preceding round of identification, and the disease progression states are used to indicate the disease progression predicted in the next stage in any round of identification. The recognition module is configured to preprocess the nth image sequence to obtain a new image; wherein the preprocessing process includes multi-angle slicing, image enhancement, and image fusion; the new image is input into the prediction model of the image recognition network for feature extraction, and the feature data output by the prediction model is used as the input of the first fully connected layer of the image recognition network; a first feature vector is obtained from the output of the first fully connected layer; when the n-1 preceding disease states and the n-1 disease progression states are in the form of feature vectors, the first feature vector, the n-1 preceding disease states, and the n-1 disease progression states are concatenated to obtain a second feature vector; the second feature vector is input into the second fully connected layer of the image recognition network; after passing through the second fully connected layer, the second feature vector is integrated into the nth disease state and the (n+1)th disease progression state of the body part in the nth round of recognition, wherein the nth disease state is used to indicate the condition of the body part in the nth round of recognition.

8. The apparatus according to claim 7, characterized in that, The first acquisition module is configured to determine M follow-up images that are temporally adjacent and most recently scanned, and to use the M follow-up images as the nth image sequence.

9. The apparatus according to claim 7, characterized in that, The prediction model includes multiple sequentially connected residual blocks; wherein each residual block includes an identity mapping and at least two convolutional layers, and the identity mapping of any residual block points from the input to the output of that residual block; or, The prediction model includes a tight-connected block; wherein, a tight-connected block includes at least two tight-connected layers; for any tight-connected block, the input of each tight-connected layer includes the outputs of all previous tight-connected layers in the tight-connected block.

10. A computer device, characterized in that, The device includes a processor and a memory, the memory storing at least one line of program code, which is loaded and executed by the processor to implement the image recognition method as claimed in any one of claims 1 to 6.

11. A computer-readable storage medium, characterized in that, The storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the image recognition method as described in any one of claims 1 to 6.

Images