Ultrasonic image acquisition method and device

Abstract

The embodiment of the invention provides an ultrasonic image acquisition method and device which are used for assisting a user in acquiring an accurate ultrasonic image. The method comprises the following steps: determining probe parameters of a target probe, target part parameters and standard section data corresponding to a target part in response to the operation that a user selects to use the target ultrasonic probe to inspect the target part; acquiring preview ultrasonic image data acquired by the target ultrasonic probe at the initial acquisition position, and acquiring motion pose data of the target ultrasonic probe at the initial acquisition position; performing data splicing on the probe parameters, the target part parameters, the standard section data, the preview ultrasonic image data and the motion pose data to obtain multi-modal data; and inputting the multi-modal data into a trained probe guide model, determining movement guide data of the target ultrasonic probe through the probe guide model, and displaying the movement guide data in a display interface, so that a user moves the target ultrasonic probe to a target acquisition position according to the movement guide data.

Description

Technical Field

[0001] The present application relates to the field of ultrasound technology, and in particular to a method and device for acquiring ultrasound images. Background Art

[0002] With the development of medical ultrasound, medical ultrasound equipment is used more and more frequently in medicine, and the use of ultrasound equipment occupies an increasingly important position in clinical medicine. For medical students, the acquisition method of ultrasound images mainly relies on teachers' lectures and demonstrations, or practice through phantoms. In practice, due to the flexibility and diversity of ultrasound sections, there is a certain gap between the actual imaging and the schematic diagrams in the textbooks. For medical students, there is not much experience in acquiring ultrasound images, and it is difficult to understand the correspondence between the probe position, image, and anatomical structure, so it is difficult to acquire accurate ultrasound images. Summary of the invention

[0003] The embodiments of the present application provide a method and device for acquiring an ultrasonic image, so as to assist a user in acquiring an accurate ultrasonic image.

[0004] In a first aspect, an embodiment of the present application provides a method for acquiring an ultrasound image, comprising:

[0005] In response to a user selecting to use a target ultrasound probe to examine a target part, determining probe parameters of the target probe, target part parameters, and standard section data corresponding to the target part;

[0006] Acquire preview ultrasound image data acquired by the target ultrasound probe at an initial acquisition position, and acquire motion posture data of the target ultrasound probe at the initial acquisition position, wherein the motion posture data is used to characterize the position and / or posture of the target ultrasound probe;

[0007] Splicing the probe parameters, the target part parameters, the standard section data corresponding to the target part, the preview ultrasound image data and the motion posture data to obtain multimodal data;

[0008] The multimodal data is input into a trained probe guidance model, movement guidance data of the target ultrasound probe is determined by the trained probe guidance model, and the movement guidance data is displayed in a display interface so that a user can move the target ultrasound probe to a target acquisition position according to the movement guidance data.

[0009] In a possible implementation, after the movement guidance data is displayed in the display interface, the method further includes: acquiring preview ultrasound image data acquired by the target ultrasound probe at the target position; determining the similarity between the preview ultrasound image data and the standard section data corresponding to the target part; when the similarity is greater than a set threshold, saving the ultrasound image data acquired by the target ultrasound probe at the target position, and displaying the target ultrasound image acquired at the target position in the display interface.

[0010] In a possible implementation, the method further includes: in response to an operation of performing image quality assessment on the target ultrasound image triggered by a user on the display interface, performing image segmentation on the target ultrasound image, and determining various structural regions in the target ultrasound image; wherein the structural region is a region where a structure included in the target part is located, and the target part includes multiple structures; and displaying the outlines of various structural regions in the target ultrasound image in the display interface, so that the user can determine the image quality of the target ultrasound image based on the number of structural regions displayed in the display interface.

[0011] In a possible implementation, the method further includes: in response to an operation of interpreting the target ultrasound image triggered by a user on the display interface, performing image segmentation on the target ultrasound image, and determining various structural regions in the target ultrasound image; wherein the structural region is a region where a structure included in the target part is located, and the target part includes multiple structures; for any structural region in the target ultrasound image, measuring the structural region according to at least one preset measurement parameter corresponding to the structural region, and obtaining measurement results corresponding to each preset measurement parameter; and displaying the measurement results corresponding to each structural region on the display interface.

[0012] In a possible implementation, performing image segmentation on the target ultrasound image to determine each structural region in the target ultrasound image includes: inputting the target ultrasound image into a trained semantic segmentation model, and obtaining the structural region of each structure in the target ultrasound image through the trained semantic segmentation model; wherein the semantic segmentation model is obtained after training with a training sample set, and the training sample set includes a plurality of standard section sample images, and each structural region in each pre-labeled standard section sample image.

[0013] In a possible implementation, obtaining the motion posture data of the target ultrasonic probe at the initial acquisition position includes: obtaining at least one sensor data collected by an inertial measurement unit located on the target ultrasonic probe; and fusing the at least one sensor data to determine the motion posture data of the target ultrasonic probe at the initial acquisition position.

[0014] In a possible implementation, the probe guidance model is trained in the following manner: M multimodal sample data corresponding to the probe are respectively input into a policy network to obtain actions that the probe needs to perform corresponding to the M multimodal sample data;

[0015] For any multimodal sample data, determine the next multimodal sample data after the probe performs the corresponding action, and the similarity information between the preview ultrasound image data collected after the probe performs the corresponding action and the reference ultrasound image data; store each multimodal sample data, the corresponding action, the similarity information and the next multimodal sample data as training samples in a cache area; input each multimodal sample data, the corresponding action and the next multimodal sample data into a value network to obtain a value function value; extract N training samples from the cache area, determine a discounted cumulative reward value according to the value function value and the similarity information of each sample, and construct a value loss function using a Monte Carlo formula according to the discounted cumulative reward value; update the value network according to the value loss function, and update the network parameters of the policy network according to the gradient information of the corresponding actions of each sample; when it is determined that the preset training conditions are met, save the network parameters of the policy network, and use the trained policy network as the probe guidance model.

[0016] In a second aspect, an embodiment of the present application provides an ultrasonic image acquisition device, comprising:

[0017] A determination module, configured to respond to a user's selection of using a target ultrasound probe to examine a target part, and determine probe parameters of the target probe, parameters of the target part, and standard section data corresponding to the target part;

[0018] an acquisition module, used to acquire preview ultrasound image data acquired by the target ultrasound probe at an initial acquisition position, and to acquire motion posture data of the target ultrasound probe at the initial acquisition position, wherein the motion posture data is used to characterize the position and / or posture of the target ultrasound probe;

[0019] A splicing module, used for splicing the probe parameters, the target part parameters, the standard section data corresponding to the target part, the preview ultrasound image data and the motion posture data to obtain multimodal data;

[0020] The determination module is further used to input the multimodal data into a trained probe guidance model, and determine the movement guidance data of the target ultrasound probe through the trained probe guidance model;

[0021] The display module is used to display the movement guidance data in a display interface so that the user can move the target ultrasound probe to a target acquisition position according to the movement guidance data.

[0022] In a third aspect, an embodiment of the present application provides an execution device, including:

[0023] A memory for storing program instructions;

[0024] The processor is used to call the program instructions stored in the memory, and execute the method described in the first aspect and different implementation methods of the first aspect according to the obtained program instructions.

[0025] In a fourth aspect, the present application is an embodiment that provides a computer-readable storage medium, wherein the computer-readable storage medium includes computer instructions, and when the computer program instructions are executed by a computer, the computer executes the method described in the first aspect and different implementation methods of the first aspect.

[0026] In the embodiment of the present application, the posture data is obtained by the inertial measurement unit located on the ultrasound probe, and multimodal data is formed with the ultrasound image and the probe / part parameters, and the multimodal data is input into the probe guidance model to determine the movement guidance data to guide the user to perform clinical ultrasound examination. The method can provide probe guidance prompts in real time according to the user's operation, so that the user can move the probe according to the probe guidance data to obtain a more ideal ultrasound image, which can assist the user to collect accurate ultrasound images. In addition, the present application combines image segmentation technology to contour the organs and lesions in the collected ultrasound images and analyze the physiological structure characteristics, so that problems can be located and diagnosed more quickly, improving the user's skill level and diagnostic accuracy, solving the problems of limited existing teaching resources, differences between classroom and clinic, and reducing learning costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0028] Figure 1 A schematic diagram of the architecture of an ultrasound-assisted teaching system provided in an embodiment of the present application;

[0029] Figure 2 A business flow chart of an ultrasound-assisted teaching system provided in an embodiment of the present application;

[0030] Figure 3 A schematic diagram of an interface of an auxiliary teaching system provided in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of the architecture of a basic teaching module provided in an embodiment of the present application;

[0032] Figure 5 A schematic diagram of a basic teaching interface provided in an embodiment of the present application;

[0033] Figure 6 A schematic diagram of an AI teaching assistant interface provided in an embodiment of the present application;

[0034] Figure 7 A business flow chart of an AI teaching assistant module provided in an embodiment of the present application;

[0035] Figure 8 A schematic diagram of the architecture of a skill testing module provided in an embodiment of the present application;

[0036] Fig. 9 A business flow chart of a skill testing module provided in an embodiment of the present application;

[0037] Fig.10 A schematic diagram of a skill test interface provided in an embodiment of the present application;

[0038] Fig.11 A schematic diagram of an ultrasonic device provided in an embodiment of the present application;

[0039] Fig.12 A flowchart of a method for acquiring an ultrasonic image provided in an embodiment of the present application;

[0040] Fig.13 A schematic diagram of an AI teaching assistant module interface provided in an embodiment of the present application;

[0041] Fig.14 A schematic diagram of an ultrasound probe provided in an embodiment of the present application;

[0042] Fig.15 A schematic diagram of a probe-guided interface provided in an embodiment of the present application;

[0043] Fig.16 A schematic diagram of another probe-guided interface provided in an embodiment of the present application;

[0044] Fig.17 A schematic diagram of another probe-guided interface provided in an embodiment of the present application;

[0045] Fig.18 A schematic diagram of an interface for section quality assessment provided in an embodiment of the present application;

[0046] Fig.19 A schematic diagram of an image interpretation interface provided in an embodiment of the present application;

[0047] Fig. 20 A schematic diagram of a training process of a probe guidance model provided in an embodiment of the present application;

[0048] Fig.21 A schematic diagram of a multi-head network structure provided in an embodiment of the present application;

[0049] Fig. 22 A schematic diagram of an ultrasound image acquisition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0052] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0053] With the development of medical ultrasound, medical ultrasound equipment is used more and more frequently in medicine, and the use of ultrasound equipment occupies an increasingly important position in clinical medicine. For medical students, the acquisition method of ultrasound images mainly relies on teachers' lectures and demonstrations, or practice through phantoms. In practice, due to the flexibility and diversity of ultrasound sections, there is a certain gap between the actual imaging and the schematic diagrams in the textbooks. For medical students, there is not much experience in acquiring ultrasound images, and it is difficult to understand the correspondence between the probe position, image, and anatomical structure, so it is difficult to acquire accurate ultrasound images.

[0054] In response to the above problems, the embodiments of the present application provide a method and device for collecting ultrasonic images. In the present application, the posture data is obtained by an inertial measurement unit located on the ultrasonic probe, and multimodal data is formed with the ultrasonic image and the probe / part parameters, and the multimodal data is input into the probe guidance model to determine the movement guidance data to guide the user to perform clinical ultrasonic examination. The method can provide probe guidance prompts in real time according to the user's operation, so that the user can move the probe according to the probe guidance data to obtain a more ideal ultrasonic image, which can assist the user to collect accurate ultrasonic images. In addition, the present application combines image segmentation technology to contour the organs and lesions in the collected ultrasonic images and analyze the physiological structure characteristics, so that the problem can be located and diagnosed more quickly, the user's skill level and diagnostic accuracy can be improved, and the existing teaching resources are limited, the classroom and the clinical differences exist. The problem of problems has been solved, and the learning cost has been reduced.

[0055] The following introduces the application scenarios of the ultrasound image acquisition method provided in the embodiments of the present application in conjunction with the accompanying drawings.

[0056] The present application embodiment provides an ultrasound-assisted teaching system, such as Figure 1 The ultrasound-assisted teaching system includes a basic teaching module, an AI teaching assistant module, and a skill testing module. Figure 2 The main business processes of the ultrasound-assisted teaching system are displayed. The default site of the ultrasound-assisted teaching system should be consistent with the current examination site as much as possible. As an example, after the user chooses to use the target ultrasound probe to examine the target site, after entering the auxiliary teaching system, the target ultrasound probe and target site are selected by default in the display interface of the auxiliary teaching system. As an example, after the user selects probe 1 and confirms that the thyroid gland should be examined, the data is transmitted to the ultrasound-assisted teaching system. After entering the teaching system, the site list on the left side of the touch screen enters the thyroid gland list by default, and probe 1 is selected. Figure 3The right side of the touch screen displays the basic teaching module, AI teaching assistant module, and skill test module. The display screen of the ultrasound system remains the scanning interface. Select a module (basic teaching module, AI teaching assistant module, skill test module) and click to enter. The touch screen displays the selected module. You can click in the module to return to the home page of the teaching system. Click Exit on the home page to exit the teaching system.

[0057] The following is a detailed introduction to the three modules included in the ultrasound-assisted teaching system.

[0058] 1. Basic teaching module

[0059] In some embodiments, the basic teaching module can provide teaching resources for normal ultrasound examination of common parts and ultrasound examination of common diseases, including anatomical structure diagrams, anatomical videos, ultrasound standard section diagrams, ultrasound standard section videos, probe scanning diagrams, probe scanning schematic videos, scanning precautions, etc., as shown in the figure. Through the basic teaching module, users can gradually master the characteristics of disease images and scanning methods on the basis of understanding and mastering normal physiological structure examinations. Figure 4 shown.

[0060] In some embodiments, after the user selects the basic teaching module, the display area of ​​the touch screen displays the basic teaching interface, such as Figure 5 The default entry point in the basic teaching module is the same as the current examination site. For example, if the current examination site is the thyroid gland, after entering the basic teaching module, the site list on the left side of the touch screen defaults to the thyroid gland list, and the right side of the touch screen displays the anatomical structure, standard ultrasound sections, probe scanning diagrams, scanning precautions, etc. Click the drop-down button "▽" to display common thyroid examination sections and common disease sections. Users can select a section to start learning. Figure 3 The display screen in the scan interface remains unchanged. Click "Back" to return to the previous level, or click "Back to Home" to return to the home page of the teaching system, and click "Exit" to exit the ultrasound-assisted teaching system.

[0061] 2. AI teaching assistant module

[0062] In some embodiments, the AI ​​teaching assistant module can provide guidance on normal ultrasound examination of common parts and ultrasound examination of common diseases, including probe guidance, section quality evaluation, interpretation of image physiological and pathological structures, and outlining, measurement and analysis, such as Figure 6 The AI ​​teaching assistant module enables users to gradually accumulate ultrasound clinical diagnosis and treatment experience based on mastering the theory.

[0063] In some scenes, Figure 7The main business process of the AI ​​teaching assistant module is displayed. After entering the module, the touch screen displays the AI ​​teaching assistant interface. The default site of the AI ​​teaching assistant is as consistent as possible with the current examination site. For example, if the current examination site is the thyroid gland, after entering the AI ​​teaching assistant module, the site list on the left side of the touch screen defaults to the thyroid gland list, and the right side of the touch screen displays probe guidance, section quality assessment, image interpretation, etc. Furthermore, the user selects a certain section to enter learning, and the corresponding learning section is displayed. When the learning is finished, you can click the return button.

[0064] 3. Skills Test Module

[0065] In some embodiments, the skill test module provides test questions on normal ultrasound examination of common parts and ultrasound examination of common diseases, including basic test, computer test, report test, etc. Figure 8 Each set of test questions is randomly selected from the system and the accuracy is evaluated, which helps students evaluate the learning effect of the theoretical and practical learning stages.

[0066] Fig. 9 The main business process of the skill test module is displayed. When the user chooses to enter the skill test module, the login interface is displayed. Enter the user name and password to enter the skill test interface. Fig.10 As shown. The default entry point of the skill test is the same as the current examination site. For example, if the current examination site is the thyroid gland, after entering the skill test, the site list on the left side of the touch screen defaults to the thyroid gland list, and the right side of the touch screen displays thyroid-related basic tests, computer tests, and report tests. The basic test mainly includes single-choice questions, multiple-choice questions, and judgment questions, which examine the user's mastery of basic knowledge; the computer test examines the user's operating techniques and evaluates the user based on the comprehensive section quality and scanning time; the report test examines the user's diagnostic ability, the standardization and professionalism of the report writing, and the ability to extract diagnostic information through the user's description of the size, location, shape, echo, etc. of the organ (or lesion). The skill test module can provide personalized learning content and suggestions based on the learning situation and progress of different users by comprehensively analyzing the user's learning records and scores, so that each user can learn at his own pace and level and better master the ultrasound scanning technology. The user enters the user name and password to log in during the test. After the test, he can view his own results, understand the weak links, and conduct targeted training. In some scenarios, users can log in as a teacher to mark test questions, and add, delete, modify, and check test questions, student information, grades, etc.

[0067] The present application embodiment provides a method for collecting ultrasound images, which can be performed by an ultrasound-assisted teaching system, and the ultrasound-assisted teaching system can be deployed in an ultrasound device. Fig.11As shown, the ultrasound device 10 includes a touch screen 101 , a display screen 102 , a processor 103 and at least one ultrasound probe 104 .

[0068] The processor 103 is configured to respond to a user's operation of selecting a target ultrasound probe in at least one ultrasound probe 104 and determine a probe parameter and a target part of the target ultrasound probe;

[0069] The touch screen 101 is used to detect the operation of selecting the AI ​​teaching assistant module of the ultrasound-assisted teaching system triggered by the user on the display interface of the touch screen 101, and display the AI ​​teaching assistant module interface in the display interface;

[0070] A target ultrasound probe in the at least one ultrasound probe 104, used to acquire an ultrasound image of an inspection section corresponding to a target part according to a user operation;

[0071] The processor 103 is used to respond to the operation of selecting a target inspection section triggered by the user on the display interface of the touch screen 101, determine the standard section data corresponding to the target inspection section of the target part; obtain the preview ultrasound image data collected by the target ultrasound probe at the initial collection position, and obtain the motion posture data of the target ultrasound probe at the initial collection position, perform data splicing on the probe parameters, the target part parameters, the standard section data corresponding to the target part, the preview ultrasound image data and the motion posture data to obtain multimodal data and input the data into the trained probe guidance model, determine the movement guidance data of the target ultrasound probe; and save the target ultrasound image data of the target inspection section according to the user's operation;

[0072] Display screen 102, in response to the user's operation of selecting a target inspection section of a target part triggered on the display interface of touch screen 101, displays a target ultrasound probe guidance interface in the display interface of display screen 102, and displays a target ultrasound image acquired by the target ultrasound probe in the display interface.

[0073] The present application embodiment provides a method for collecting ultrasound images, see Fig.12 As shown, the method can be performed by an ultrasonic device 10, and the specific process is as follows:

[0074] 1201 , in response to a user selecting to use a target ultrasound probe to examine a target part, determine probe parameters of the target probe, target part parameters, and standard section data corresponding to the target part.

[0075] In some embodiments, different ultrasound probes in the ultrasound device can be used to perform ultrasound examinations on different parts of the human body. When the user determines the target part to be examined, the corresponding target ultrasound probe can be selected according to the target part. As an example, the ultrasound device includes 4 ultrasound probes. When the user determines that an ultrasound examination of the thyroid gland is required, the ultrasound probe among the 4 ultrasound probes can be determined to be used for ultrasound examination of the thyroid gland. For example, ultrasound probe 1 can perform ultrasound examination of the thyroid gland, and the user can select ultrasound probe 1.

[0076] In some embodiments, after the user selects a target ultrasound probe, the probe parameters of the target ultrasound probe may be determined according to the target ultrasound probe selected by the user. For example, after the user selects ultrasound probe 1, the probe parameters corresponding to ultrasound probe 1 may be obtained.

[0077] Further, in response to the user selecting to use the target ultrasound probe to examine the target part, the touch screen displays the AI ​​teaching assistant module interface. As an example, the AI ​​teaching assistant module interface is as follows: Fig.13 As shown, the left side of the display interface shows the human body parts that can be examined, such as thyroid, breast, musculoskeletal, gynecology and obstetrics. The right side of the display interface shows the ultrasound probes corresponding to the human body parts to be examined, such as probe 1, probe 2, probe 3 and probe 4. In addition, the right side of the display interface also includes probe guidance, section quality assessment and image interpretation.

[0078] In some scenarios, the default entry point in the AI ​​teaching assistant module is consistent with the current examination part. Fig.13 In the example, the target part is the thyroid gland, and the ultrasound probe is selected as probe 1, which is consistent with the target part and target ultrasound probe selected by the user above.

[0079] Furthermore, in response to the user's operation of selecting a target inspection section, standard section data corresponding to the target part is determined.

[0080] For example, Fig.13 As shown in the figure, the target part selected by the user is the thyroid gland, and the ultrasound probe selects probe 1. When the user selects the probe guide drop-down button "▽", common thyroid examination sections and common disease sections are displayed, such as the cross section, longitudinal section, sagittal section, nodule, hyperthyroidism and thyroiditis, etc. Among them, nodule, hyperthyroidism and thyroiditis represent the sections corresponding to the disease. When the user selects the cross section, the standard section data corresponding to the thyroid cross section is determined.

[0081] 1202, obtaining preview ultrasound image data acquired by the target ultrasound probe at the initial acquisition position, and obtaining motion posture data of the target ultrasound probe at the initial acquisition position.

[0082] In some embodiments, obtaining the motion posture data of the target ultrasonic probe at the initial acquisition position can be achieved by: obtaining at least one sensor data collected by an inertial measurement unit located on the target ultrasonic probe. Further, the motion posture data of the target ultrasonic probe at the initial acquisition position can be obtained by processing based on the at least one sensor data. The motion posture data is used to characterize the position and / or posture of the target ultrasonic probe.

[0083] As an example, in this embodiment, the inertial measurement unit (IMU) is built into the ultrasonic probe, such as Fig.14 As shown. Optionally, the inertial measurement unit includes sensors such as accelerometers, gyroscopes, and magnetometers. In some scenarios, after the user places the target ultrasonic probe at the initial acquisition position, the sensor data collected by the multiple sensors included in the inertial measurement unit on the target ultrasonic probe can be obtained. Furthermore, the sensor data measured by the inertial measurement unit can be processed and fused to obtain the position and / or attitude information of the target ultrasonic probe at the initial acquisition position. Among them, the attitude information may include the speed, displacement, yaw angle, pitch angle, roll angle, etc. of the target ultrasonic probe.

[0084] In some scenarios, after the user places the target ultrasound probe at the initial acquisition position, the user may also obtain preview ultrasound image data acquired by the target ultrasound probe at the initial acquisition position.

[0085] 1203 , the probe parameters, the target part parameters, the standard section data corresponding to the target part, the preview ultrasound image data and the motion posture data are spliced ​​to obtain multimodal data.

[0086] In some scenarios, different data can be preprocessed and the preprocessed data can be spliced ​​to obtain multimodal data.

[0087] Exemplarily, N frames of preview ultrasound images may be spliced ​​as ultrasound image information. Further, the ultrasound image information is normalized to obtain ultrasound image features, the standard section data corresponding to the target part is normalized to obtain standard section features, and the motion posture data is normalized to obtain motion posture features. Oneshot encoding is used for the probe / part parameters to obtain probe / part parameter features, and the ultrasound image features, standard section features, motion posture features, and probe / part parameter features are spliced ​​to obtain multimodal data.

[0088] 1204 , input the multimodal data into the trained probe guidance model, determine the movement guidance data of the target ultrasound probe through the trained probe guidance model, and display the movement guidance data in the display interface.

[0089] In some embodiments, after the user selects the target inspection section on the touch screen display interface, the probe guidance interface is displayed on the display screen display interface. Fig.15 As shown, the display interface of the display screen includes a probe guidance prompt, and the preview ultrasound image collected by the probe at the initial position is displayed in the middle. In some scenarios, the user can use the target ultrasound probe to start scanning, and the motion guidance data is calculated based on the multimodal data by the probe guidance model and displayed in the probe guidance prompt area of ​​the display interface of the display screen, so that the user can move the target ultrasound probe to the target acquisition position according to the motion guidance data.

[0090] In some scenarios, after the probe guidance data is determined by the probe guidance model, the probe guidance data can be converted into text or voice to facilitate human-computer interaction. Fig.15 As shown, the probe guidance data displayed in the probe guidance prompt area is "move the probe clockwise in the upper left corner". The user can move the target ultrasound probe according to the probe guidance data displayed in the probe guidance prompt area to move the target ultrasound probe to the target acquisition position.

[0091] In some embodiments, after the movement guidance data is displayed in the display interface of the display screen, preview ultrasound image data collected by the target ultrasound probe at the target position can also be obtained, and further, the similarity between the preview ultrasound image data and the standard section data corresponding to the target part can be determined. When the similarity is greater than a set threshold, the ultrasound image data collected by the target ultrasound probe at the target position is saved, and the target ultrasound image collected at the target position is displayed in the display interface. Fig.16 As shown, the left side of the display interface of the display screen can also display the preview ultrasound image score, anatomical structure, and standard section, and the right side displays the saved target ultrasound image. In some embodiments, after the target ultrasound probe acquires the preview ultrasound image at the target position, the similarity between the preview ultrasound image data and the standard section data can be calculated, and the preview ultrasound image score can be determined based on the similarity. Among them, the similarity between the preview ultrasound image data and the standard section data can be determined by a quality assessment model, and the similarity between the preview ultrasound image data and the standard section data can also be called an image quality score.

[0092] In a possible implementation, the greater the similarity between the preview ultrasound image data and the standard section data, the higher the score of the preview ultrasound image. When the similarity between the preview ultrasound image data and the standard section data is greater than a set threshold, the ultrasound image data acquired by the target ultrasound probe at the target position is saved, and the target ultrasound image acquired at the target position is displayed in the display interface of the display screen. For example, when the similarity between the preview ultrasound image data and the standard section data is greater than 80%, the ultrasound image data acquired by the target ultrasound probe at the target position is saved, and the saved ultrasound image is displayed on the right side of the display interface of the display screen.

[0093] Optionally, when the similarity between the preview ultrasound image data and the standard section data is greater than a set threshold, a prompt box indicating that the image quality has reached the standard can be displayed on the display interface to prompt the user to save the target ultrasound image. Fig.17 shown.

[0094] Optionally, the user can hold the target ultrasound probe at the target position for 2-3 seconds without moving to automatically save the target ultrasound image. Alternatively, the user can save the target ultrasound image by pressing the save key. In some scenarios, after the target ultrasound image is saved, the user can be reminded by voice that the target ultrasound image has been saved. As an example, after the target ultrasound image is saved, a "beep" prompt sound can be used to remind the user that the image has been saved successfully.

[0095] In some scenarios, when the user moves the target ultrasound probe to the target position and the similarity between the preview ultrasound image data at the target position and the standard section data is less than or equal to a set threshold, the preview ultrasound image at the target position is input into the probe guidance model, and the next probe guidance data of the target ultrasound probe is determined and moved until the similarity between the preview ultrasound image data and the standard section data is greater than the set threshold.

[0096] As an example, when the user selects the target ultrasound probe and starts scanning, the probe guidance prompt part displays the probe posture, and the user places the probe on the corresponding part of the patient according to the prompt. The scanning steps are as follows:

[0097] 1. The display screen displays the preview ultrasound image collected by the target ultrasound probe at the current position. The quality assessment model evaluates the current image quality score. If the image quality score is less than the set threshold, step 2 is executed; if the image quality score is greater than the set threshold, step 4 is entered.

[0098] 2. The probe guidance model receives the multimodal data, outputs the probe guidance data and displays it in the probe guidance prompt area of ​​the display interface of the display screen.

[0099] 3. The user moves the target ultrasound probe according to the probe guidance data. At this time, the preview ultrasound image acquired by the probe in the display area in the middle of the display screen changes, and step 1 is executed.

[0100] 4. When the image quality score is greater than the set threshold, a prompt box "Image quality has reached the standard" will pop up on the display interface to remind the user to save the ultrasound image.

[0101] In some embodiments, in response to an operation of evaluating the image quality of a target ultrasound image triggered by a user on a display interface of a touch screen, the target ultrasound image is segmented to determine various structural regions in the target ultrasound image. The structural region is a region where a structure included in a target part is located, and the target part includes multiple structures. Furthermore, the outlines of various structural regions in the target ultrasound image may be displayed in a display interface of the display screen, so that the user can determine the image quality of the target ultrasound image based on the number of outlines of the structural regions displayed in the display interface of the display screen.

[0102] Exemplarily, in response to a user clicking on the touch screen display, Fig.13 After clicking the section quality assessment drop-down button "▽" in the AI ​​teaching assistant interface shown, the various section types corresponding to the saved ultrasound images are displayed. In response to the user's selection of a certain section, the display screen enters the section quality assessment interface. For example, when the user selects a cross-section on the touch screen display interface, the display screen enters the cross-section quality assessment interface. In the display screen display interface, the left column displays the standard section, quality assessment score, and display structure of the thyroid cross-section from top to bottom; the current section is displayed in the middle, and the structure is depicted; the saved image is displayed on the right. For example, the quality assessment score is determined by calculating the similarity between the saved cross-section and the standard cross-section. Furthermore, the outlines of each structural area can be displayed in the display interface according to the segmentation results of the cross-section, such as Fig.18 Further, the structures included in the cross section can be compared with the multiple structures included in the standard cross section, and the existing structures or missing structures can be displayed in the structure display part in the left column.

[0103] In some embodiments, in response to an operation of interpreting a target ultrasound image triggered by a user on a touch screen display interface, the target ultrasound image is segmented to determine various structural regions in the target ultrasound image. Further, any of the structural regions may be processed according to at least one preset processing algorithm corresponding to any of the structural regions in the target ultrasound image to determine at least one measurement result corresponding to any of the structural regions, and at least one measurement result corresponding to each of the structural regions may be displayed on the display screen display interface.

[0104] As an example, in response to a user clicking on a touch screen such as Fig.13 After pressing the image interpretation drop-down button "▽" in the AI ​​teaching assistant interface shown, the various section types corresponding to the saved ultrasound images are displayed. Exemplarily, after the user selects the nodule section on the touch screen, the ultrasound image corresponding to the nodule section is interpreted, and the image interpretation result is displayed on the display screen display interface. Specifically, the ultrasound image corresponding to the nodule section can be segmented to determine the nodule structure area. Furthermore, the nodule structure area is processed according to the segmentation result using at least one preset processing algorithm to determine the measurement result corresponding to the nodule structure area. For example, the nodule area corresponds to six preset processing algorithms: aspect ratio, edge, echo, calcification, boundary, and grading. The nodule area is processed by the six preset processing algorithms respectively to obtain measurement results, and the measurement results are displayed in the left area of ​​the display screen display interface, and the measurement results are displayed on the image and in the middle area of ​​the display screen display interface, such as Fig.19 This interface can increase the user's readability and facilitate the user's intuitive understanding of the pathological structure.

[0105] In some embodiments, performing image segmentation on a target ultrasound image and determining each structural region in the target ultrasound image can be achieved by: inputting the target ultrasound image into a semantic segmentation model, and outputting the structural region of each structure in the target ultrasound image through the semantic segmentation model. The semantic segmentation model is obtained after training with a training sample set, and the training sample set includes a standard section image corresponding to the target ultrasound image and the structural region of each structure in the standard section image.

[0106] In some embodiments, the probe guidance model can be trained in the following manner, see Fig. 20 , the specific process is as follows:

[0107] 2001, the M multimodal sample data corresponding to the probe are respectively input into the strategy network to obtain the actions corresponding to the M multimodal sample data.

[0108] In some scenarios, the network parameters π of the policy network can be initialized first θ and the network parameter V of the value network ω Among them, the policy network structure can be realized by a multi-head action network. The network structure of the multi-head action network can adopt a multi-layer convolutional neural network, plus a fully connected layer, and then connected to n fully connected layers. Each action is controlled by a separate action output head, so each dimension can have an action output, such as Fig.21 shown.

[0109] Furthermore, each of the M multimodal sample data can be input into the strategy network to obtain actions corresponding to the M multimodal sample data. The multimodal sample data includes probe parameters, standard section data corresponding to the target part, preview ultrasound image data and motion posture data.

[0110] In the embodiment, the output of the policy network is the probability corresponding to the action of each dimension, and the action corresponding to the multimodal sample data is selected according to the probability distribution. As an example, assuming that the action includes three dimensions, the action corresponding to the maximum probability value in the action of each dimension is selected to obtain the actions corresponding to the three dimensions. For example, including rotation angle, displacement and tilt angle, the rotation angle includes multiple angles corresponding to clockwise rotation, and multiple angles of counterclockwise rotation, the displacement includes moving forward 1 cm, 2 cm, moving backward 1 cm, 2 cm, etc., and the tilt angle includes tilting 30 degrees to the upper left, tilting 30 degrees to the upper right, tilting 30 degrees to the left rear, and tilting 30 degrees to the right rear, etc. As an example, the output of the policy network is the probability corresponding to the rotation angle, displacement and tilt angle of multiple actions respectively. When the maximum probability value in the dimension corresponding to the rotation angle is 30 degrees clockwise rotation, the maximum probability value in the dimension corresponding to the displacement is 1 cm forward movement, and the maximum probability value in the dimension corresponding to the rotation angle is 30 degrees upper left tilt, then the selected action is 30 degrees clockwise rotation, 1 cm forward movement and 30 degrees upper left tilt.

[0111] 2002. For each multimodal sample data, determine the next multimodal sample data after the corresponding action is performed, and the similarity information between the preview ultrasound image data and the reference ultrasound image data collected after the probe performs the corresponding action.

[0112] In some scenarios, the multimodal sample data may be acted upon and the next multimodal sample data after the act is performed may be determined, and similarity information between the preview ultrasound image data and the reference ultrasound image data (i.e., standard section data) after the multimodal sample data is acted upon may be determined. In some embodiments, the similarity information may be represented by a reward value.

[0113] 2003. Each multimodal sample data, the corresponding action, similarity information and the next multimodal sample data are stored in a buffer as training samples.

[0114] As an example, the state transfer process data (s t ,a t ,r t ,s t+1 ) is stored in the buffer as a sample. t Represents multimodal sample data, a trepresents the action determined by the output of the policy network, r t Represents similarity information (reward value), s t+1 Indicates t Execute a t The next multimodal sample data after .

[0115] 2004, each multimodal sample data and the corresponding action and next multimodal sample data are input into the value network to obtain the value function value corresponding to each initial multimodal data.

[0116] 2005, extract N samples from the buffer area, determine the discounted cumulative reward value according to the value function value and the reward value, and construct the value loss function using the Monte Carlo formula according to the discounted cumulative reward value.

[0117] 2006, the value network is updated according to the value loss function, and the network parameters of the policy network are updated according to the gradient of the sample corresponding to the action.

[0118] In some embodiments, the Actor network parameters are updated according to the following formula:

[0119]

[0120] In some embodiments, the discount cumulative reward value satisfies the conditions described in the following formula:

[0121]

[0122] The Monte Carlo formula is:

[0123]

[0124] 2007. When it is determined that the preset training conditions are met, the network parameters of the policy network are saved, and the trained policy network is used as a probe to guide the model.

[0125] Based on the same technical concept, see Fig. 22 As shown, the embodiment of the present application provides an ultrasound image acquisition device 2200, which can perform any step in the ultrasound image acquisition method described above. To avoid repetition, the device 2200 is not described here. The device 2200 includes a determination module 2201, an acquisition module 2202, a splicing module 2203 and a display module 2204.

[0126] The determination module 2201 is used to respond to the user's selection of using a target ultrasound probe to examine a target part, and to determine the probe parameters of the target probe, the target part parameters, and the standard section data corresponding to the target part;

[0127] An acquisition module 2202 is used to acquire preview ultrasound image data acquired by the target ultrasound probe at an initial acquisition position, and acquire motion posture data of the target ultrasound probe at the initial acquisition position, wherein the motion posture data is used to characterize the position and / or posture of the target ultrasound probe;

[0128] A splicing module 2203 is used to splice the probe parameters, the target part parameters, the standard section data corresponding to the target part, the preview ultrasound image data and the motion posture data to obtain multimodal data;

[0129] The determination module 2201 is further configured to input the multimodal data into a trained probe guidance model, and determine the movement guidance data of the target ultrasound probe through the trained probe guidance model;

[0130] The display module 2204 is used to display the movement guidance data in a display interface, so that the user can move the target ultrasound probe to a target acquisition position according to the movement guidance data.

[0131] In some embodiments, after the movement guidance data is displayed in the display interface, the acquisition module 2202 is further used to:

[0132] Acquire preview ultrasound image data collected by the target ultrasound probe at the target position;

[0133] The determination module 2201 is further used to determine the similarity between the preview ultrasound image data and the standard section data corresponding to the target part;

[0134] When the similarity is greater than a set threshold, saving the ultrasonic image data collected by the target ultrasonic probe at the target position;

[0135] The display module 2204 is used to display the target ultrasound image acquired at the target position in the display interface.

[0136] In some embodiments, the determination module 2201 is further used to respond to an operation of performing image quality assessment on the target ultrasound image triggered by the user on the display interface, perform image segmentation on the target ultrasound image, and determine various structural regions in the target ultrasound image; wherein the structural region is a region where a structure included in the target part is located, and the target part includes multiple structures;

[0137] The display module 2204 is used to display the outlines of each structural region in the target ultrasound image in the display interface, so that the user can determine the image quality of the target ultrasound image according to the number of structural regions displayed in the display interface.

[0138] In some embodiments, the determination module 2201 is further used to respond to an operation of interpreting the target ultrasound image triggered by the user on the display interface, perform image segmentation on the target ultrasound image, and determine various structural regions in the target ultrasound image; wherein the structural region is a region where a structure included in the target part is located, and the target part includes multiple structures;

[0139] For any structural region in the target ultrasound image, measuring the structural region according to at least one preset measurement parameter corresponding to the structural region, and obtaining measurement results corresponding to each preset measurement parameter;

[0140] The display module 2204 is used to display the measurement results corresponding to each structural area on the display interface.

[0141] In some embodiments, the determination module 2201, when performing image segmentation on the target ultrasound image to determine each structural region in the target ultrasound image, is specifically used to:

[0142] Inputting the target ultrasound image into a trained semantic segmentation model, and obtaining a structural region of each structure in the target ultrasound image through the trained semantic segmentation model;

[0143] The semantic segmentation model is obtained after training with a training sample set, wherein the training sample set includes a plurality of standard section sample images and various structural regions in each pre-marked standard section sample image.

[0144] In some embodiments, the acquisition module 2202, when acquiring the motion posture data of the target ultrasound probe at the initial acquisition position, is specifically used to:

[0145] Acquiring at least one sensor data collected by an inertial measurement unit located on the target ultrasonic probe;

[0146] The at least one sensor data is fused and processed to determine the motion posture data of the target ultrasound probe at the initial acquisition position.

[0147] In some embodiments, the determination module 2201 is further configured to train the probe guidance model in the following manner:

[0148] Inputting M initial multimodal data into the policy network respectively to obtain actions corresponding to the M initial multimodal data respectively;

[0149] For each initial multimodal data, determine the next multimodal data and reward value after executing the corresponding action;

[0150] Storing each of the initial multimodal data, the action, the reward value, and the next multimodal data as samples in a buffer;

[0151] Input each of the initial multimodal data and the corresponding action into the value network to obtain the value function value corresponding to each of the initial multimodal data;

[0152] Extracting N samples from the buffer area, determining a discounted cumulative reward value according to the value function value and the reward value, and constructing a value loss function using a Monte Carlo formula according to the discounted cumulative reward value;

[0153] Update the value network according to the value loss function, and update the network parameters of the strategy network according to the gradient of the action corresponding to the sample;

[0154] When the preset training requirements are met, the network parameters of the policy network are saved, and the trained policy network is used as the probe guidance model.

[0155] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0156] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0159] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for collecting ultrasonic images, characterized in that: include: In response to a user selecting to use a target ultrasound probe to examine a target part, determining probe parameters of the target probe, target part parameters, and standard section data corresponding to the target part; Acquire preview ultrasound image data acquired by the target ultrasound probe at an initial acquisition position, and acquire motion posture data of the target ultrasound probe at the initial acquisition position, wherein the motion posture data is used to characterize the position and / or posture of the target ultrasound probe; Splicing the probe parameters, the target part parameters, the standard section data corresponding to the target part, the preview ultrasound image data and the motion posture data to obtain multimodal data; The multimodal data is input into a trained probe guidance model, movement guidance data of the target ultrasound probe is determined by the trained probe guidance model, and the movement guidance data is displayed in a display interface so that a user can move the target ultrasound probe to a target acquisition position according to the movement guidance data.

2. The method according to claim 1, characterized in that After displaying the movement guidance data in the display interface, the method further includes: Acquire preview ultrasound image data collected by the target ultrasound probe at the target position; Determining the similarity between the preview ultrasound image data and the standard section data corresponding to the target part; When the similarity is greater than a set threshold, the ultrasonic image data acquired by the target ultrasonic probe at the target position is saved, and the target ultrasonic image acquired at the target position is displayed in the display interface.

3. The method according to claim 2, characterized in that The method further comprises: In response to an operation of performing image quality assessment on the target ultrasound image triggered by a user on the display interface, the target ultrasound image is segmented to determine various structural regions in the target ultrasound image; wherein the structural region is a region where a structure included in the target part is located, and the target part includes a plurality of structures; The outlines of the various structural regions in the target ultrasound image are displayed in the display interface, so that the user can determine the image quality of the target ultrasound image according to the number of structural regions displayed in the display interface.

4. The method according to claim 2, characterized in that The method further comprises: In response to an operation of interpreting the target ultrasound image triggered by a user on the display interface, the target ultrasound image is segmented to determine various structural regions in the target ultrasound image; wherein the structural region is a region where a structure included in the target part is located, and the target part includes a plurality of structures; For any structural region in the target ultrasound image, measuring the structural region according to at least one preset measurement parameter corresponding to the structural region, and obtaining measurement results corresponding to each preset measurement parameter; The display interface displays the measurement results corresponding to each of the structural regions.

5. The method according to claim 3 or 4, characterized in that The performing image segmentation on the target ultrasound image to determine each structural region in the target ultrasound image includes: Inputting the target ultrasound image into a trained semantic segmentation model, and obtaining a structural region of each structure in the target ultrasound image through the trained semantic segmentation model; The semantic segmentation model is obtained after training with a training sample set, wherein the training sample set includes a plurality of standard section sample images and various structural regions in each pre-marked standard section sample image.

6. The method according to claim 1, characterized in that The acquiring the motion posture data of the target ultrasound probe at the initial acquisition position includes: Acquiring at least one sensor data collected by an inertial measurement unit located on the target ultrasonic probe; The at least one sensor data is fused and processed to determine the motion posture data of the target ultrasound probe at the initial acquisition position.

7. The method according to claim 1, characterized in that The probe guidance model is trained as follows: Inputting M multimodal sample data corresponding to the probe into the strategy network respectively, and obtaining the actions that the probe needs to perform corresponding to the M multimodal sample data respectively; For any multimodal sample data, determine the next multimodal sample data after the probe performs the corresponding action, and similarity information between the preview ultrasound image data and the reference ultrasound image data acquired after the probe performs the corresponding action; Storing each multimodal sample data, the corresponding action, the similarity information and the next multimodal sample data as training samples in a buffer area; Input each multimodal sample data, the corresponding action, and the next multimodal sample data into the value network to obtain the value function value; Extracting N training samples from the buffer area, determining a discounted cumulative reward value according to the value function value of each sample and the similarity information, and constructing a value loss function using a Monte Carlo formula according to the discounted cumulative reward value; Update the value network according to the value loss function, and update the network parameters of the strategy network according to the gradient information of the actions corresponding to each sample; When it is determined that the preset training conditions are met, the network parameters of the policy network are saved, and the trained policy network is used as the probe guidance model.

8. An ultrasonic image acquisition device, characterized in that: include: A determination module, configured to respond to a user's selection of using a target ultrasound probe to examine a target part, and determine probe parameters of the target probe, target part parameters, and standard section data corresponding to the target part; an acquisition module, used to acquire preview ultrasound image data acquired by the target ultrasound probe at an initial acquisition position, and to acquire motion posture data of the target ultrasound probe at the initial acquisition position, wherein the motion posture data is used to characterize the position and / or posture of the target ultrasound probe; A splicing module, used for splicing the probe parameters, the target part parameters, the standard section data corresponding to the target part, the preview ultrasound image data and the motion posture data to obtain multimodal data; The determination module is further used to input the multimodal data into a trained probe guidance model, and determine the movement guidance data of the target ultrasound probe through the trained probe guidance model; The display module is used to display the movement guidance data in a display interface so that the user can move the target ultrasound probe to a target acquisition position according to the movement guidance data.

9. An execution device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 7 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises computer program instructions, and when the computer program instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 7.

Images