Image recognition method and device, equipment and medium

Abstract

The invention discloses an image recognition method and device, equipment and a medium. The method comprises the steps that an image set composed of ultrasonic images of the lung of a target person is acquired, and the image set comprises at least one ultrasonic image of at least one lung partition of the target person; determining the type of at least one ultrasonic image in the image set; performing sign recognition on at least one ultrasonic image in the ultrasonic images of which the types are determined; predicting the state of the lung of the target person according to the recognition result of at least one ultrasonic image in the ultrasonic images subjected to sign recognition; at least one target image and the prediction result are output, the corresponding recognition result is shown in the output target image, and the target image is an ultrasonic image used for predicting the state of the lung of the target person.

Description

Technical Field

[0001] The present invention relates to the field of computers, and particularly to an image recognition method, apparatus, device, and medium. Background Art

[0002] Medical images play a huge role in daily medical work. Among them, medical ultrasonic imaging is a commonly used type of medical image. Medical ultrasonic imaging refers to a technology for exploring tissues and lesions in the human body through ultrasonic waves, mainly relying on the ultrasonic waves emitted by an ultrasonic probe to carry the echoes of different tissues in the human body to construct an internal ultrasonic image of the human body. Ultrasonic imaging has been widely used due to its advantages such as no electromagnetic radiation, providing real-time imaging, and non-invasive exploration.

[0003] Common medical ultrasonic imaging modes include: B-mode ultrasonic image, M-mode ultrasonic image, Doppler color ultrasonic image, etc. Based on ultrasonic images of different modes, different signs can be recognized. In the related art, when performing ultrasonic exploration on the human lungs, it is usually mainly by identifying whether the B-mode ultrasonic image of the human lungs contains B lines (B Line) and some related attributes of the B lines to predict the state of the human lungs. Summary of the Invention

[0004] In the related art, the state of the human lungs is predicted by identifying whether the B-mode ultrasonic image of the human lungs contains B lines (B Line) and some related attributes of the B lines. However, because the prediction only relies on this one sign of the B line, the prediction result may be inaccurate. The embodiments of the present application provide an image recognition method, apparatus, device, medium, chip, and computer program product, which can, to a certain extent, solve the technical problem that the prediction result of predicting the state of the human lungs through the B-mode ultrasonic image of the human lungs in the related art may be inaccurate.

[0005] The first aspect of the embodiments of the present application provides an image recognition method, and the method includes:

[0006] Obtain an image set composed of ultrasonic images of the lungs of a target person, where the image set includes at least one ultrasonic image of at least one lung region of the target person;

[0007] Determine the type of at least one ultrasonic image in the image set;

[0008] Perform sign recognition on at least one ultrasonic image in the ultrasonic images of the determined type;

[0009] Predict the state of the lungs of the target person according to the recognition results of at least one ultrasonic image in the ultrasonic images on which sign recognition has been performed;

[0010] Output at least one target image and a prediction result, and show the corresponding recognition result in the output target image, where the target image is an ultrasound image for predicting the state of the lungs of the target person.

[0011] A second aspect of the embodiments of the present application provides an image recognition device, which includes:

[0012] A first acquisition module, configured to acquire an image set composed of ultrasound images of the lungs of a target person, where the image set includes at least one ultrasound image of at least one lung region of the target person;

[0013] A first determination module, configured to determine the type of at least one ultrasound image in the image set;

[0014] A first recognition module, configured to perform sign recognition on at least one ultrasound image in the ultrasound images of the determined type;

[0015] A first scoring module, configured to score the recognition result of at least one ultrasound image in the ultrasound images on which sign recognition has been performed according to a preset scoring rule;

[0016] A first prediction module, configured to predict the state of the lungs of the target person according to the scores of at least one ultrasound image in the scored ultrasound images;

[0017] A first output module, configured to output at least one target image and a prediction result, and show the corresponding recognition result and score in the output target image, where the target image is an ultrasound image for predicting the state of the lungs of the target person.

[0018] A third aspect of the embodiments of the present application provides an electronic device, which includes: a processor and a memory, where the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the image recognition method described in the first aspect are implemented.

[0019] A fourth aspect of the embodiments of the present application provides a readable storage medium, where a program or instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps of the image recognition method described in the first aspect are implemented.

[0020] A fifth aspect of the embodiments of the present application provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or instruction to implement the steps of the image recognition method described in the first aspect.

[0021] A sixth aspect of the embodiments of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the steps of the image recognition method described in the first aspect.

[0022] In the embodiments of the present application, in the task of predicting the lung status based on the ultrasound image of the lungs of a target person, it is possible to flexibly identify signs other than B-lines in the ultrasound image of the lungs of not only one type of imaging mode, and based on this, predict the lung status of the target person. Compared with the related art that only predicts the lung status of a human body through the B-line signs in the B-mode ultrasound image of the human lungs, the embodiments of the present application can dynamically combine the signs identified in the ultrasound images of other types of imaging modes to predict the lung status of the target person, so that the dimension of the signs on which the prediction process is based is more, and thus the prediction result is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a schematic diagram of the twelve-division of the human lungs;

[0024] Figure 2 is a schematic diagram of the step flow of a picture recognition method provided by the embodiments of the present application;

[0025] Figure 3 is Figure 2 a schematic diagram of the sub-step flow of step S13 shown in

[0026] Figure 4 is Figure 2 a schematic diagram of the sub-step flow of step S13 shown in

[0027] Figure 5 is Figure 2 a schematic diagram of the sub-step flow of step S13 shown in

[0028] Figure 6 is Figure 1 a schematic diagram of the sub-step flow of step S14 shown in

[0029] Figure 7 is Figure 6 a schematic diagram of the sub-step flow of step S142 shown in

[0030] Figure 8 is a schematic diagram of the structure of the image recognition device provided by the embodiments of the present application;

[0031] Figure 9 is a schematic diagram of the structure of the electronic device provided by the embodiments of the present application;

[0032] Figure 10It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0033] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0034] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are usually of the same category, and the number of objects is not limited. For example, the first object can be one or multiple. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the associated objects before and after.

[0035] For the convenience of correctly understanding the embodiments of the present application, the following first defines and explains the technical terms that may be involved in the embodiments of the present application. Unless otherwise specified hereinafter, the semantics of the corresponding technical terms follow the definitions herein:

[0036] A computer or computing device refers to all electronic devices based on the Turing computability theory, von Neumann architecture or Harvard architecture. For example, mobile phones, smart watches, single-chip microcomputer systems, etc. are all computers, not limited to the narrow sense of computers such as personal computers (PCs, Personal Computers) mentioned in daily life.

[0037] An application, that is, a computer application program (Computer Program or application software), is written in certain programming languages and runs on a certain target architecture computer system. A computer application program refers to a coded instruction sequence that can be executed by a device with information processing capabilities such as a computer to obtain a certain result, or a symbolic instruction sequence or symbolic statement sequence that can be automatically converted into a coded instruction sequence.

[0038] To facilitate those skilled in the art to understand the embodiments provided by the present application as a whole, the following specifically describes the technical defects in the related technologies targeted by the embodiments of the present application.

[0039] Lung ultrasound examination is a non-invasive medical imaging method used to assist in observing the human lungs for medical staff or relevant personnel to refer to for medical activities. In the related art, the lung state of a human body is usually predicted based on the B-line signs identified from B-mode ultrasound images, but the accuracy rate of the prediction results is often low. The following exemplarily illustrates this error.

[0040] The B-line sign is an abnormal sign, that is, the B-line can only be identified in the B-mode ultrasound image detected when there is a lesion in the human lungs. In actual lung exploration, it is usually necessary to perform zonal monitoring on the left and right lungs of the human body. As Figure 1 shown, Figure 1 is a schematic diagram of the twelve zones of the human lungs. Please refer to Figure 1 , in order to make the prediction results as accurate as possible, it is usually necessary to scan the twelve zones shown in Figure 1 . However, in actual exploration, the B-line cannot be identified in the B-mode ultrasound image of each zone of the lungs. For example, the B-line may be identified in the B-mode ultrasound image of zone 1 shown in Figure 1 , but the B-line is not identified in zone 7 shown in Figure 1 . However, there is actually pleural effusion in zone 7 of the lungs, and in this case, relying solely on the lung state predicted by the B-line sign will result in serious errors. Therefore, the method of predicting the lung state of a human body solely based on the B-line signs identified from B-mode ultrasound images is relatively one-sided and may not accurately predict the lung state of a human body in some cases.

[0041] An image recognition method, device, equipment, storage medium, chip, and computer program product provided by an embodiment of the present application can effectively solve the above technical problems. The following is an explanation with reference to the drawings.

[0042] As Figure 2 shown, Figure 2 is a schematic diagram of the step flow of a picture recognition method provided by an embodiment of the present application. The execution subject of this image recognition method can be a computer, an ultrasonic exploration device, etc. As Figure 2 shown, the picture recognition method includes the following steps:

[0043] S11, obtaining an image set composed of ultrasound images of the lungs of a target person, where the image set includes at least one ultrasound image of at least one lung zone of the target person.

[0044] The target person is any person whose lung state needs to be predicted.

[0045] The ultrasound (Lung Ultrasound) image of the lungs is the image scanned by an ultrasound transducer (or probe) on the lungs.

[0046] The type of ultrasound image in the image set can be an A-mode ultrasound image, a B-mode ultrasound image, an M-mode ultrasound image, or even a (Doppler) color flow ultrasound image. The image set can include ultrasound images of partial lung regions of the target person, or can also include ultrasound images of all lung regions of the target person. For a certain lung region of the target person, the corresponding ultrasound image may not be included in the image set, or one corresponding ultrasound image may be included, or multiple corresponding ultrasound images may be included. For a certain lung region of the target person, at least one type of ultrasound image among the A-mode ultrasound image, B-mode ultrasound image, M-mode ultrasound image, and color flow ultrasound image of this region can be included.

[0047] The ultrasound images in the image set can be those stored in a storage device or storage medium after being pre-scanned for the lungs of the target person, and are read by the execution entity performing this image recognition method from the storage device or storage medium.

[0048] The ultrasound images in the image set can also be obtained in real time by the execution entity performing this image recognition method from the ultrasonic probe end. Affected by the scanning position of the ultrasonic probe (also called a transducer), during the process of scanning the lungs, the probe can be horizontal (parallel to the ribs) or vertical (perpendicular to the ribs). For example, during horizontal scanning, the probe can be placed between two ribs for scanning horizontal lung ultrasound images, and at this time the image does not contain acoustic shadows or artifacts introduced by rib occlusion; during vertical scanning, the probe needs to scan across two ribs for imaging, and at this time the lung ultrasound image is affected by the ribs resulting in artifacts or acoustic shadows in the imaging area; affected by the shape and size of the probe, there may be no influence of rib acoustic shadows or artifacts in the transverse and longitudinal scans of some probes, such as phased array probes. During the scanning process of the left and right lungs, the two, three, four, and six-zone scans are respectively used. During the scanning process, after one zone scan is completed, it can be automatically switched to the next zone, and the zone switching also supports user customization.

[0049] S12, determine the type of at least one ultrasound image in the image set.

[0050] As mentioned above, the type of ultrasound image can be an A-mode ultrasound image, a B-mode ultrasound image, an M-mode ultrasound image, or even a (Doppler) color flow ultrasound image. A pre-set image type recognition algorithm can be called to recognize the type of at least one ultrasound image in the image set, that is, to recognize whether the imaging mode of the ultrasound image is A-mode, B-mode, M-mode, or color Doppler.

[0051] S13, perform sign recognition on at least one ultrasound image among the ultrasound images of the determined type.

[0052] For an ultrasonic image in an ultrasonic image for which the imaging mode has been determined, a pre-set sign recognition algorithm matching its type can be called to perform sign recognition on it to identify possible signs contained therein. For example, for a determined B-mode ultrasonic image, a pre-set first recognition algorithm can be called to identify the first signs in the B-mode ultrasonic image.

[0053] In some alternative embodiments, the image set includes B-mode ultrasonic images of each lung sub-region in the at least one lung sub-region; as Figure 3 shown, Figure 3 is Figure 2 a schematic diagram of the sub-step process of step S12 shown in. Please refer to Figure 3 , for a determined B-mode ultrasonic image, S13 includes:

[0054] S131, calling a pre-set first recognition algorithm to perform recognition of the first signs on at least one B-mode ultrasonic image;

[0055] S132, in the case of recognizing the first signs, determining the relevant attributes of the recognized first signs.

[0056] The first recognition algorithm is a pre-set algorithm for performing first sign recognition on B-mode ultrasonic images. The first signs include at least one of the following: A-line (A Line, parallel and equally spaced hyperechoic lines under the pleural line), B-line (in lung ultrasound examination, an abnormal sign in the lung, and the B-line affects the prediction of the lung state), pleural line (Pleural Line, the first hyperechoic line in lung ultrasound), lung consolidation, atelectasis, air bronchogram, lung lesions, pleural effusion.

[0057] In the case of recognizing the first signs from a certain B-mode ultrasonic image, the relevant attributes of all the first signs recognized from the image are determined. The relevant attributes of the first signs are some indicators for quantifying the first signs.

[0058] For example, when B-lines are recognized in a certain B-mode ultrasound image, determine the position of the recognized B-lines in the lungs, calculate the effective area of the B-lines, the type of B-lines (fused B-lines and non-fused B-lines), the B-line distance (B Line distance, the emission point of the B-line is the pleural line, and the physical distance between two B-lines at the emission point position), the scanning method of the B-mode ultrasound image, and the position of the intercostal space in the B-mode ultrasound image, etc. It should be noted that different probes and different scanning methods will have a certain impact on the calculation of the effective area of the B-lines, and special attention should be paid. When A-lines are recognized in a certain B-mode ultrasound image, determine the position of the recognized A-lines in the lungs, the distance between A-lines, etc. When the pleural line is recognized in a certain B-mode ultrasound image, determine the position of the recognized pleural line in the lungs, the size and thickness of the pleural line, etc. When lung consolidation is recognized in a certain B-mode ultrasound image, determine the area size, perimeter, and area of the recognized lung consolidation, etc. When the air bronchogram is recognized in a certain B-mode ultrasound image, determine the position, quantity, size, perimeter, and area of the recognized air bronchogram, etc. When lung lesions are recognized in a certain B-mode ultrasound image, determine the position, quantity, size, perimeter, and area of the recognized lung lesions, etc. When the air bronchogram is recognized in a certain B-mode ultrasound image, determine the position, size, and fluid volume of the recognized pleural effusion, etc. For example, the area of the pleural line can be obtained through an identification algorithm, and the maximum value of the column sum in the area is used as the calculated value of the pleural line thickness. For example, it can also be determined whether the pleura is continuous. If the pleural line is a closed and complete area, it is continuous; if the pleural line is not a closed and complete area, it is discontinuous.

[0059] In some alternative embodiments, the image set further includes M-mode ultrasound images of each lung sub-region in the at least one lung sub-region; as Figure 4 shown, Figure 4 is Figure 1 a schematic diagram of the sub-step process of step S13 shown in Figure 3 . Please refer to

[0060] S133, when the first sign includes the pleural line, call a preset second recognition algorithm to recognize the second sign for at least one co-region M-mode ultrasound image; the co-region M-mode ultrasound image is the M-mode ultrasound image of the lung sub-region that is the same as the lung sub-region corresponding to the B-mode ultrasound image where the pleural line is recognized;

[0061] S134, when the second sign includes the pleural line, determine the relevant attributes of the same pleural line in the first sign and the second sign according to the pleural line in the first sign and the pleural line in the second sign.

[0062] The second recognition algorithm is a pre-set algorithm for recognizing the second signs in M-mode ultrasound images. The second signs include at least one of the following: pleural line, pleural sliding sign, seashore sign, curtain sign, and lung fluctuation sign.

[0063] In the case where the pleural line is recognized from a certain B-mode ultrasound image, the second recognition algorithm can be called to recognize the second signs in the co-region M-mode ultrasound image corresponding to the B-mode ultrasound image. In the case where the recognized second signs include the pleural line, based on the pleural line in the first signs and the pleural line in the second signs, the relevant attributes of the same pleural line in the first signs and the second signs are comprehensively determined. For example, information such as the position of the recognized same pleural line in the lungs, the size and thickness of the pleural line is determined. By fusing the information of the pleural line recognized from the B-mode ultrasound image with the information of the pleural line recognized from the M-mode ultrasound image, and jointly determining the relevant attributes of the recognized pleural line based on both, the accuracy of the recognized pleural line is made higher, and further improving the prediction accuracy of the lung condition of the target person.

[0064] In some optional embodiments, for the determined M-mode ultrasound image, S13 includes:

[0065] Calling the pre-set second recognition algorithm to recognize the second signs in at least one M-mode ultrasound image, and in the case where the second signs are recognized, determining the relevant attributes of the recognized second signs.

[0066] For example, in the case where the pleural sliding sign is recognized from the M-mode ultrasound image, the pleural sliding frequency can be determined, etc. In the case where the pleural sliding sign is recognized from the M-mode ultrasound image, the width of the seashore sign can be determined, etc.

[0067] In some optional embodiments, the image set includes color Doppler ultrasound images of each lung sub-region in the at least one lung sub-region; as Figure 5 shown, Figure 5 is Figure 2 a schematic diagram of the sub-step process of step S13 shown in Figure 5 . For the determined color Doppler ultrasound image, S13 includes:

[0068] S135, calling the pre-set third recognition algorithm to recognize the blood flow pattern in at least one color Doppler ultrasound image.

[0069] The third recognition algorithm is a pre-set algorithm for recognizing the blood flow pattern in color Doppler ultrasound images. The blood flow patterns include: abnormal blood flow pattern (in a dendritic shape) and normal blood flow pattern (not in a dendritic shape).

[0070] In some alternative embodiments, the first signs identified from B-mode ultrasound images, the second signs identified from M-mode ultrasound images, and the blood flow patterns identified from color Doppler ultrasound images can be combined to comprehensively predict the benignancy and malignancy of abnormal signs.

[0071] For example, in the case where a lesion is identified from a B-mode ultrasound image, the benignancy and malignancy of the lesion can be predicted based on the edge flatness of the lesion determined from the B-mode image, in combination with the blood flow pattern identified from the color Doppler ultrasound image (and the state of the pleural line identified from the M-mode ultrasound image).

[0072] The above-mentioned first recognition algorithm, second recognition algorithm, and third recognition algorithm can be machine learning methods, such as convolutional neural networks, deep learning-based classification algorithms ResNet, VGG, or traditional image processing techniques. For example, due to the special morphology and structure of B-lines, B-lines can be identified by analyzing the vertical features of the image, and then quantitative analysis indicators such as the position, quantity, and spacing of B-lines can be calculated.

[0073] In the embodiments of the present application, the first recognition algorithm, second recognition algorithm, and third recognition algorithm can be the same algorithm or different algorithms. For example, they can be deep learning-based recognition algorithms such as ResNet, UNet, or machine learning algorithms based on image denoising, enhancement, image feature extraction, and combination with classifiers; image denoising algorithms, enhancement algorithms, feature extraction algorithms, etc. For example, the machine learning algorithm preprocesses the image, including image denoising and enhancement, extracts key feature information such as edge contours and gray values, and then identifies the type of the image or the type of key pixel points in the image through an SVM classifier.

[0074] S14. Predict the state of the lungs of the target person according to the recognition results of at least one ultrasound image in the ultrasound images for which signs have been recognized.

[0075] In the embodiments of the present application, after signs are recognized for some images in the image set, the state of the lungs of the target person can be preliminarily predicted according to the recognition results of the ultrasound images for which signs have been recognized. For example, in the case where pleural effusion is identified from the ultrasound image, it can be preliminarily predicted that the lungs of the target person have developed lesions.

[0076] In some alternative embodiments, the state of the lungs of the target person is predicted based on a preset scoring rule. As Figure 6 shown, Figure 6 is Figure 2 a schematic diagram of the sub-step process of step S14 shown in Figure 6 , S14 includes:

[0077] S141. Score the recognition results of at least one ultrasound image in the ultrasound images with signs recognized according to a preset scoring rule.

[0078] S142. Predict the state of the lungs of the target person according to the scores of at least one ultrasound image in the scored ultrasound images.

[0079] In some alternative embodiments, the scoring rule is as follows:

[0080] If the recognition result is: the pleural line is continuous and the number of single-region B-lines is less than 3; then the score (for this partition) is 0 points;

[0081] If the recognition result is: the number of B-lines is more than 3, the range of fused B-lines accounts for less than 50% of the intercostal section, and the area of the solid lesion < 1 cm2; if any of the above is met, then (for this partition) record 1 point;

[0082] If the recognition result is: the range of fused B-lines accounts for more than 50% of the intercostal section, the area of the solid lesion is 1 cm2 - 2 cm2, if any of the above is met, then (for this partition) record 2 points;

[0083] If the recognition result is: the area of the solid lesion > 2 cm 2 , dynamic bronchial sign, pleural effusion; if any of the above is met (for this partition) record 3 points.

[0084] This scoring rule can be for each lung partition or for each ultrasound image.

[0085] The recognition results of at least one ultrasound image in the ultrasound images with signs recognized can be scored based on the exemplary scoring rule. After scoring, predict the state of the lungs of the target person according to the scores of at least one ultrasound image in the scored ultrasound images. Exemplarily, it is set that when the score of a certain lung partition (the average of the scores of all ultrasound images of this partition) exceeds a set threshold, it is predicted that the lungs of the target person have a lesion.

[0086] In some alternative embodiments, score each lung partition. After scoring all twelve lung partitions, predict the state of the lungs of the target person. As Figure 7 shown, Figure 7 is Figure 6 a schematic diagram of the sub-step process of step S142 shown in Figure 7 , S142 includes:

[0087] S1421. Determine the scores of each lung partition in the twelve lung partitions of the lungs of the target person according to at least one ultrasound image in the scored ultrasound images.

[0088] Determine the lung sub-region to which each ultrasound image in at least one of the identified ultrasound images belongs, and score the lung sub-region based on the recognition results of all the ultrasound images of the lung sub-region.

[0089] In some embodiments, the score of a lung sub-region is the mean of the scores of all the ultrasound images of that sub-region.

[0090] In some embodiments, when analyzing multiple consecutive lung ultrasound images, a representative target image can be automatically selected for the final result display. For example, the target image includes an image selected from the B-mode ultrasound images that meets a preset condition, and the preset condition includes at least one of the following: the selected ultrasound image is the image with the largest B-line area identified among all the B-mode ultrasound images corresponding to the lung sub-region, and the selected ultrasound image is the image with the largest number of B-lines identified among all the B-mode ultrasound images corresponding to the lung sub-region. Use the sign recognition result of this target image as the scoring basis for the corresponding lung sub-region.

[0091] S1422, determine the total score of the lungs of the target person according to the scores of each lung sub-region in the twelve lung sub-regions of the target person;

[0092] In the embodiments of the present application, the total score of the lungs of the target person is the sum of the scores of each lung sub-region in the twelve lung sub-regions.

[0093] S1423, predict the lung status of the target person according to the total score.

[0094] Exemplarily, it is set that when the total score of the lungs exceeds a set threshold, it is predicted that the lungs of the target person have a lesion.

[0095] S15, output at least one target image and the prediction result, and show the corresponding recognition result in the output target image, where the target image is an ultrasound image used to predict the lung status of the target person.

[0096] Output the target image and the prediction result on a display device, and show the corresponding recognition result in the output target image so that medical staff can observe and understand the lung status of the target person.

[0097] In some alternative embodiments, the score of the output target image can also be output synchronously.

[0098] In some alternative embodiments, all types of target images of the same lung sub-region are output simultaneously in the same interface, and the corresponding recognition results are shown in the output target images. For example, for a sub-region of the lungs of a target person, the B-mode target ultrasound image, M-mode target ultrasound image, and color Doppler target ultrasound image of the lung sub-region can be output simultaneously, so that medical staff can compare and observe the lung status of the target person.

[0099] For different types of ultrasound images, the analysis results of the lung ultrasound images are shown in whole or in part by combining the analysis results of different signs; for example, in an image containing B-lines in lung ultrasound, the quantitative analysis results such as the position, type, spacing, and intercostal space of the B-lines are shown. For lung ultrasound examinations carried out on the same person at different time periods, it is supported to use the time axis as the horizontal axis and each quantitative index as the vertical axis to form a trend analysis chart in the form of a bar chart, column chart, table, text, number, or a combination of one or more of the above forms; the vertical axis of the trend analysis chart can also be the quantitative analysis results of the same sub-region; for example, the vertical axis can be a bar statistical chart of the number of B-lines in the first sub-region of the left lung. For the clinical meanings represented by different signs in lung ultrasound, it is supported to describe the high or low score of the sub-region or the severity of lung abnormalities in the form of colors, text, numbers, etc.; for example: red can be used to represent a relatively high lung score value in the current sub-region, such as 3 points; light red can be used to represent a relatively low lung score value in the current sub-region, such as 2 points; yellow can be used to represent a relatively small lung score in the current sub-region, such as 1 point; green can be used to represent a lung score of 0 in the current sub-region.

[0100] In the embodiments of the present application, in the task of predicting the lung status based on the ultrasound images of the lungs of a target person, the lung ultrasound images of imaging modes not limited to one type can be flexibly recognized for signs not limited to B-lines, and based on this, the lung status of the target person can be predicted. Compared with the related art that only predicts the lung status of a human body through the B-line signs in the B-mode ultrasound image of the human lungs, the embodiments of the present application can dynamically combine the signs recognized in the ultrasound images of other types of imaging modes to predict the lung status of the target person, so that the dimension of the signs on which the prediction process is based is more, and thus the prediction result is more accurate.

[0101] For the image recognition method provided by the embodiments of the present application, the execution subject can be an image recognition device. In the embodiments of the present application, taking the image recognition device executing the image recognition method as an example, the image recognition device provided by the embodiments of the present application is described.

[0102] As Figure 8 shown, a schematic structural diagram of an image recognition device provided by the embodiments of the present application is shown. Please refer to Figure 8 The image recognition device 5 includes:

[0103] The first acquisition module 51 is configured to acquire an image set composed of ultrasound images of the lungs of a target person, where the image set includes at least one ultrasound image of at least one lung partition of the target person;

[0104] The first determination module 52 is configured to determine the type of at least one ultrasound image in the image set;

[0105] The first recognition module 53 is configured to perform sign recognition on at least one ultrasound image among the ultrasound images of the determined type;

[0106] The first prediction module 54 is configured to predict the state of the lungs of the target person according to the recognition results of at least one ultrasound image among the ultrasound images on which sign recognition has been performed;

[0107] The first output module 55 is configured to output at least one target image and a prediction result, and show the corresponding recognition result in the output target image, where the target image is an ultrasound image for predicting the state of the lungs of the target person.

[0108] In some alternative embodiments, the type includes: B-mode ultrasound images; the image set includes B-mode ultrasound images of each lung partition in the at least one lung partition; the first recognition module 53 includes:

[0109] The first calling sub-module is configured to call a preset first recognition algorithm to perform recognition of a first sign on at least one B-mode ultrasound image; the first sign includes at least one of the following: A-line, B-line, pleural line, pulmonary consolidation, atelectasis, air bronchogram, pulmonary lesions, pleural effusion;

[0110] The first determination sub-module is configured to determine the relevant attributes of the recognized first sign when the first sign is recognized.

[0111] In some alternative embodiments, the type further includes: M-mode ultrasound images; the image set further includes M-mode ultrasound images of each lung partition in the at least one lung partition; the first recognition module 53 further includes:

[0112] The second calling sub-module is configured to call a preset second recognition algorithm to perform recognition of a second sign on at least one co-region M-mode ultrasound image when the first sign includes a pleural line; the co-region M-mode ultrasound image is an M-mode ultrasound image of the same lung partition as the lung partition corresponding to the B-mode ultrasound image on which the pleural line is recognized; the second sign includes at least one of the following: pleural line, pleural sliding sign, seashore sign, curtain sign, lung fluctuation sign;

[0113] A second determination sub-module, configured to, when the second signs identified include a pleural line, determine relevant attributes of the same pleural line in the first signs and the second signs according to the pleural line in the first signs and the pleural line in the second signs.

[0114] In some alternative embodiments, the type includes: color Doppler ultrasound images; the image set includes color Doppler ultrasound images of each lung sub-region in the at least one lung sub-region; the first recognition module 53 includes:

[0115] A third call sub-module, configured to call a preset third recognition algorithm to perform blood flow pattern recognition on at least one color Doppler ultrasound image.

[0116] In some alternative embodiments, the first prediction module 54 includes:

[0117] A first scoring sub-module, configured to score the recognition results of at least one ultrasound image in the ultrasound images on which signs have been recognized according to a preset scoring rule;

[0118] A first prediction sub-module, configured to predict the state of the lungs of the target person according to the scores of at least one ultrasound image in the scored ultrasound images.

[0119] In some alternative embodiments, the first prediction sub-module includes:

[0120] A first determination unit, configured to determine the scores of each lung sub-region in the twelve lung sub-regions of the lungs of the target person according to at least one ultrasound image in the scored ultrasound images;

[0121] A second determination unit, configured to determine the total score of the lungs of the target person according to the scores of each lung sub-region in the twelve lung sub-regions of the lungs of the target person;

[0122] A third prediction unit, configured to predict the state of the lungs of the target person according to the total score.

[0123] In some alternative embodiments, the image recognition device 5 further includes:

[0124] A second output module, configured to output the scores of the target images to be output.

[0125] In some alternative embodiments, the first output module 55 includes:

[0126] A first output sub-module, configured to simultaneously output all types of target images of the same lung sub-region in the same interface, and show the corresponding recognition results in the target images to be output.

[0127] In some alternative embodiments, the target image includes an image selected from B-mode ultrasound images that meets a preset condition, and the preset condition includes at least one of the following: the selected ultrasound image is the image with the largest area of the B-lines identified among all B-mode ultrasound images corresponding to this lung partition, and the selected ultrasound image is the image with the largest number of B-lines identified among all B-mode ultrasound images corresponding to this lung partition.

[0128] The image recognition device 5 in the embodiments of the present application may be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal or other devices other than a terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a Mobile Internet Device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. It may also be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The embodiments of the present application do not make specific limitations.

[0129] The image recognition device in the embodiments of the present application may be a device with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems. The embodiments of the present application do not make specific limitations.

[0130] The image recognition device 5 provided in the embodiments of the present application can implement Figures 2 to 7 each process implemented by the method embodiments. To avoid repetition, it will not be elaborated here.

[0131] In some alternative embodiments, as Figure 9 shown, the embodiments of the present application further provide an electronic device 130, including a processor 131 and a memory 132. A program or instruction that can run on the processor 131 is stored on the memory 132. When the program or instruction is executed by the processor 131, it implements each step of the above image recognition method embodiment and can achieve the same technical effect. To avoid repetition, it will not be elaborated here.

[0132] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0133] Figure 10 Schematic diagram of the hardware structure of an electronic device for implementing an embodiment of the present application.

[0134] The electronic device 140 includes, but is not limited to: a radio frequency unit 141, a network module 142, an audio output unit 143, an input unit 144, a sensor 145, a display unit 146, a user input unit 147, an interface unit 148, a memory 149, and a processor 1410 and other components. Those skilled in the art can understand that the electronic device 140 may further include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 1410 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. Figure 10 The structure of the electronic device shown in does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0135] Among them, the processor 1410 is used for:

[0136] Obtain an image set composed of ultrasonic images of the lungs of a target person, where the image set includes at least one ultrasonic image of at least one lung region of the target person;

[0137] Determine the type of at least one ultrasonic image in the image set;

[0138] Perform sign recognition on at least one ultrasonic image among the ultrasonic images of the determined type;

[0139] Predict the state of the lungs of the target person according to the recognition results of at least one ultrasonic image among the ultrasonic images on which sign recognition has been performed;

[0140] Output at least one target image and a prediction result, and show the corresponding recognition result in the output target image, where the target image is an ultrasonic image for predicting the state of the lungs of the target person.

[0141] It should be understood that in the embodiments of the present application, the input unit 144 may include a Graphics Processing Unit (GPU) 1441 and a microphone 1442. The GPU 1441 processes the image data of static pictures or videos obtained by an image capturing device (such as a camera) in the video capture mode or the image capture mode. The display unit 146 may include a display panel 1461, and the display panel 1461 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 147 includes at least one of a touch panel 1471 and other input devices 1472. The touch panel 1471 is also referred to as a touch screen. The touch panel 1471 may include two parts: a touch detection device and a touch controller. The other input devices 1472 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be elaborated here.

[0142] The memory 149 can be used to store software programs and various data. The memory 149 mainly includes a first storage area for storing programs or instructions and a second storage area for storing data. Among them, the first storage area can store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 149 may include a volatile memory or a non-volatile memory, or the memory 149 may include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash memory. The volatile memory may be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 149 in the embodiments of the present application includes, but is not limited to, these and any other suitable types of memories.

[0143] The processor 1410 may include one or more processing units; optionally, the processor 1410 integrates an application processor and a modem processor, where the application processor mainly processes operations related to the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor may not be integrated into the processor 1410 either.

[0144] In the image recognition method provided by the embodiments of the present application, the execution subject may be a vehicle. In the embodiments of the present application, taking the vehicle as an example to execute the image recognition method, the electronic system provided by the embodiments of the present application is described.

[0145] Any of the above product embodiments can implement each process of the above image recognition method embodiment through the operation of its own processor, and can achieve the same technical effects. To avoid repetition, they will not be described one by one.

[0146] The embodiments of the present application further provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, it implements each process of the above image recognition method embodiment and can achieve the same technical effects. To avoid repetition, it will not be described here. Wherein, the processor is the processor in the electronic device or electronic system described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory ROM, random access memory RAM, magnetic disk or optical disc, etc.

[0147] The embodiments of the present application further provide a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement each process of the above image recognition method embodiment and can achieve the same technical effects. To avoid repetition, it will not be described here.

[0148] It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, system chip, chip system, or system-on-chip, etc.

[0149] The embodiments of the present application provide a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement each process of the above image recognition method embodiment and can achieve the same technical effects. To avoid repetition, it will not be described here.

[0150] In the implementation manners provided by the embodiments of the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device implementation manners described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0151] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this implementation manner.

[0152] In addition, in each implementation manner of the embodiments of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0153] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the methods described in each implementation manner of the embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0154] The above is only the implementation manner of the embodiments of the present application, and does not limit the patent scope of the embodiments of the present application. The above specific implementation manner is merely illustrative rather than restrictive. Under the inspiration of the present application, those of ordinary skill in the art, by making equivalent structural or equivalent process transformations using the content of the specification and drawings of the embodiments of the present application, or directly or indirectly applying them in other related technical fields, without departing from the purpose of the present application and the scope protected by the claims, can still make many forms, all of which are similarly included in the patent protection scope of the embodiments of the present application.

Claims

1. An image recognition method, characterized in that, The method includes: Obtaining an image set composed of ultrasonic images of the lungs of a target person, where the image set includes at least one ultrasonic image of at least one lung partition of the target person; Determining the type of at least one ultrasonic image in the image set; Performing sign recognition on at least one ultrasonic image among the ultrasonic images of the determined type; Predicting the state of the lungs of the target person according to the recognition results of at least one ultrasonic image among the ultrasonic images on which sign recognition has been performed; Outputting at least one target image and a prediction result, and showing the corresponding recognition result in the output target image, where the target image is an ultrasonic image for predicting the state of the lungs of the target person.

2. The method according to claim 1, characterized in that, The type includes: B-mode ultrasonic image; the image set includes B-mode ultrasonic images of each lung partition in the at least one lung partition; the performing sign recognition on at least one ultrasonic image among the ultrasonic images of the determined type includes: Invoking a preset first recognition algorithm to perform recognition of a first sign on at least one B-mode ultrasonic image; the first sign includes at least one of the following: A-line, B-line, pleural line, pulmonary consolidation, atelectasis, air bronchogram, pulmonary lesion, pleural effusion; In the case where a first sign is recognized, determining the relevant attributes of the recognized first sign.

3. The method according to claim 2, wherein The type further includes: M-mode ultrasonic image; the image set further includes M-mode ultrasonic images of each lung partition in the at least one lung partition; the performing sign recognition on at least one ultrasonic image among the ultrasonic images of the determined type further includes: In the case where the pleural line is included in the recognized first sign, invoking a preset second recognition algorithm to perform recognition of a second sign on at least one co-region M-mode ultrasonic image; the co-region M-mode ultrasonic image is an M-mode ultrasonic image of the same lung partition as the lung partition corresponding to the B-mode ultrasonic image on which the pleural line is recognized; the second sign includes at least one of the following: pleural line, pleural sliding sign, seashore sign, curtain sign, lung fluctuation sign; In the case where the pleural line is included in the recognized second sign, determining the relevant attributes of the same pleural line in the first sign and the second sign according to the pleural line in the first sign and the pleural line in the second sign.

4. The method according to claim 1, characterized in that, The type includes: color Doppler ultrasound image; the image set includes color Doppler ultrasound images of each lung partition in the at least one lung partition; the performing sign recognition on at least one ultrasonic image among the ultrasonic images of the determined type includes: Invoking a preset third recognition algorithm to perform blood flow pattern recognition on at least one color Doppler ultrasound image.

5. The method according to claim 1, wherein The predicting the state of the lungs of the target person according to the recognition results of at least one ultrasonic image among the ultrasonic images on which sign recognition has been performed includes: Scoring the recognition results of at least one ultrasonic image among the ultrasonic images on which sign recognition has been performed according to a preset scoring rule; Predicting the state of the lungs of the target person according to the scores of at least one ultrasonic image among the scored ultrasonic images.

6. The method according to claim 5, characterized in that The predicting the state of the lungs of the target person according to the scores of at least one ultrasonic image among the scored ultrasonic images includes: Determine the scores of each lung sub-region in the twelve lung sub-regions of the target person's lungs according to at least one ultrasound image in the scored ultrasound images; Determine the total score of the target person's lungs according to the scores of each lung sub-region in the twelve lung sub-regions of the target person's lungs; Predict the state of the target person's lungs according to the total score.

7. The method according to claim 5, wherein The method further includes: Output the score of the output target image.

8. The method according to any one of claims 1 to 7, characterized in that, The output of at least one target image and the prediction result, and showing the corresponding recognition result in the output target image, includes: Output all types of target images of the same lung sub-region in the same interface, and show the corresponding recognition result in the output target image.

9. The method according to claim 1, characterized in that, The target image includes an image selected from B-mode ultrasound images that meets a preset condition, and the preset condition includes at least one of the following: the selected ultrasound image is the image with the largest area of the B-lines recognized in all B-mode ultrasound images corresponding to this lung sub-region, and the selected ultrasound image is the image with the largest number of B-lines recognized in all B-mode ultrasound images corresponding to this lung sub-region.

10. An image recognition device, characterized in that, The device includes: A first acquisition module, configured to acquire an image set composed of ultrasound images of the lungs of a target person, where the image set includes at least one ultrasound image of at least one lung sub-region of the target person; A first determination module, configured to determine the type of at least one ultrasound image in the image set; A first recognition module, configured to perform sign recognition on at least one ultrasound image in the ultrasound images of which the type has been determined; A first prediction module, configured to predict the state of the lungs of the target person according to the recognition result of at least one ultrasound image in the ultrasound images on which sign recognition has been performed; A first output module, configured to output at least one target image and a prediction result, and show the corresponding recognition result in the output target image, where the target image is an ultrasound image used to predict the state of the lungs of the target person.

11. An electronic device, characterized in that, The electronic device includes: a processor and a memory, where the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the image recognition method according to any one of claims 1 to 9 are implemented.

12. A readable storage medium, characterized in that, A program or instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps of the image recognition method according to any one of claims 1 to 9 are implemented.

Images