Ultrasound credentialing system

Abstract

To provide systems and methods for automated ultrasound credentialing.SOLUTION: A credentialing system for issuing a sonographer credential to a sonography candidate is provided, the credentialing system comprising: an ultrasound system configured to generate ultrasound images; and a candidate credentialing application implemented at least partially in hardware of the credentialing system and configured to generate image quality scores for a first subset of the ultrasound images and communicate, based on the image quality scores, a second subset of the ultrasound images to a reviewer computing device.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The embodiments disclosed herein relate to ultrasonic systems. More specifically, the embodiments disclosed herein relate to ultrasonic qualification systems.

Background Art

[0002] Generally, an ultrasonic system transmits sound waves at frequencies above the audible spectrum into the body, receives echo signals generated by the reflection of the sound waves from internal body parts, and generates an ultrasonic image by converting the echo signals into electrical signals for image generation. Since those ultrasonic systems are non-invasive and can provide immediate imaging results, ultrasonic systems are widely used in care facilities. In most of these care facilities, the demand for qualified ultrasonic operators far exceeds the number of available qualified ultrasonic operators.

[0003] This difference in the demand and supply of certified ultrasound operators is due in part to the extensive training and review processes of the certification systems used by care facilities to certify ultrasound operators. For example, candidates (e.g., students) training to become certified ultrasound operators often perform ultrasound examinations on various patients with various medical conditions and are required to submit ultrasound data (e.g., imaging results) for manual review and approval by reviewers in the care facility. This review process usually cannot provide real-time feedback to candidates during the ultrasound examination because reviewers cannot review the ultrasound data of each candidate during the ultrasound examination. This problem worsens, for example, when multiple candidates are generating ultrasound examination data simultaneously for review based on different patients within a care facility. Thus, candidates may not receive feedback from reviewers until well after the ultrasound examination, such as several days or weeks. Thus, the ultrasound examination may not be memorable to the candidate when the candidate receives reviewer feedback, and as a result, the candidate may not be able to fully utilize the feedback, which may slow down the candidate's certification process.

[0004] Traditional certification systems are necessarily subjective and often inaccurate because they rely on human judgment. This is because reviewers may judge that ultrasound data is acceptable even when it is not, and vice versa. Furthermore, traditional certification systems are generally not standardized among reviewers (within a care facility and / or across different care facilities), so one reviewer may judge that ultrasound data is unacceptable while another reviewer may judge that the same ultrasound data is acceptable. Also, the dependence on reviewers (e.g., one or more senior clinicians) places a significant burden on care facilities in terms of the time required for reviewers to review submissions from candidates and generally oversee the certification program.

[0005] Furthermore, in the case of an ultrasound examination that requires urgent treatment (e.g., determination of free fluid in a patient), the candidate may not be able to submit ultrasound data for review based on an examination of a living patient. In these cases, the qualification system can generate simulated ultrasound images via a training or simulator system, and the candidate can submit the simulated ultrasound images to the reviewers of the qualification system. However, since reviewers typically recognize that the pseudo-ultrasound images are not real ultrasound images, they have an inherent bias when grading the pseudo-ultrasound images. Therefore, the reviewer may grade the pseudo-ultrasound images as acceptable because the pseudo-ultrasound images are not real ultrasound images but pseudo-images.

[0006] Therefore, conventional ultrasound qualification systems cause delays in the qualification process, impose a burden on care facilities, and are inadequately trained, yet ultrasound operators may still be qualified. Therefore, patients who require ultrasound examinations may receive care that is inferior to the best available care. SUMMARY OF THE INVENTION

[0007] A first embodiment is a qualification system for issuing an ultrasound examiner qualification to an ultrasound examination candidate, the qualification system including an ultrasound system configured to generate ultrasound images, and a candidate qualification application at least partially implemented on the hardware of the qualification system, the candidate qualification application generating an image quality score for a first subset of the ultrasound images and configured to communicate a second subset of the ultrasound images to a reviewer computing device based on the image quality score.

[0008] A second embodiment is the qualification system of the first embodiment, further comprising a reviewer computing device.

[0009] A third embodiment is the qualification system of the first embodiment, wherein the first subset and the second subset are different from each other.

[0010] In the fourth embodiment, the candidate qualification determination application determines guidance for a candidate for ultrasonic examination to improve at least one of the image quality scores based on the fact that at least one of the image quality scores is below a threshold score, and further includes a display device that displays the guidance in a visual representation. It is a qualification determination system according to the first embodiment.

[0011] In the fifth embodiment, the visual representation includes at least one of a training video, an icon of an ultrasonic probe, an arrow indicating the direction in which the ultrasonic probe is to be moved, and an icon of the orientation of a grip for holding the ultrasonic probe. It is a qualification determination system according to the fourth embodiment.

[0012] In the sixth embodiment, the candidate qualification determination application is implemented to communicate a second subset of ultrasonic images to a reviewer computing device based on the fact that the percentage of at least one of the image quality scores exceeds a threshold score, and the first subset is one or more of the ultrasonic images of at least one first anatomical structure, at least one ultrasonic image generated in a first imaging mode of the ultrasonic system, or any of at least one ultrasonic image generated according to a first examination protocol. It is a qualification determination system according to the first embodiment.

[0013] In the seventh embodiment, the candidate qualification determination application implements a neural network for generating an image quality score. It is a qualification determination system according to the first embodiment.

[0014] In the eighth embodiment, the neural network is implemented to generate an image quality score based on at least one of the orientation of the grip of the ultrasonic probe of the ultrasonic system, the amount of movement of the ultrasonic probe, the amount of movement of the candidate for ultrasonic examination, the audio content, the magnitude of the pressure applied to the patient by the ultrasonic probe, and the amount of time spent by the candidate for ultrasonic examination to operate the ultrasonic system to generate an ultrasonic image. It is a qualification determination system according to the seventh embodiment.

[0015] The ninth embodiment is a qualification system of the first embodiment configured such that a candidate qualification application receives ultrasonic data from an ultrasonic probe, generates an image quality score based on the ultrasonic data, and generates an ultrasonic examination score based on the image quality score.

[0016] The tenth embodiment is a qualification system of the first embodiment in which the second subset includes any one of one or more of the ultrasonic images of at least one second anatomical structure, at least one ultrasonic image generated in a second imaging mode, or at least one ultrasonic image generated according to a second examination protocol.

[0017] The eleventh embodiment is a computer-implemented method for issuing an ultrasonic examiner qualification to an ultrasonic examination candidate, the method including generating an ultrasonic image by a processor, generating an image quality score for a first subset of the ultrasonic images by the processor, and communicating a second subset of the ultrasonic images to a reviewer computing device by the processor based on the image quality score.

[0018] The twelfth embodiment is the method of the eleventh embodiment, further including communicating the image quality score to a reviewer computing device by a processor.

[0019] The thirteenth embodiment is the method of the eleventh embodiment, in which the first subset and the second subset are different from each other.

[0020] The fourteenth embodiment is the method of the eleventh embodiment, further including determining guidance for an ultrasonic examination candidate to improve at least one of the image quality scores based on at least one of the image quality scores being below a threshold score by a processor, and displaying the guidance in a visual representation.

[0021] The 15th embodiment is the method of the 14th embodiment, where the visual representation includes at least one of a training video, an icon of an ultrasonic probe, an arrow indicating the direction in which the ultrasonic probe is moved, and an icon of the orientation of a grip for holding the ultrasonic probe.

[0022] The 16th embodiment further includes communicating, by a processor, a second subset of ultrasonic images to a reviewer computing device based on at least one percentage of image quality scores exceeding a threshold score, where the first subset includes one or more of ultrasonic images of at least one first anatomical structure, at least one ultrasonic image generated in a first imaging mode of an ultrasonic system, or any of at least one ultrasonic image generated according to a first examination protocol, which is the method of the 11th embodiment.

[0023] The 17th embodiment is the method of the 11th embodiment, where the processor implements a neural network for generating an image quality score.

[0024] The 18th embodiment is the method of the 17th embodiment, where the neural network is implemented to generate an image quality score based on at least one of the orientation of the grip of the ultrasonic probe of the ultrasonic system, the amount of movement of the ultrasonic probe, the amount of movement of the ultrasonic examination candidate, the audio content, the magnitude of the pressure applied to the patient by the ultrasonic probe, and the amount of time spent by the ultrasonic examination candidate to operate the ultrasonic system to generate an ultrasonic image.

[0025] The 19th embodiment is the method of the 11th embodiment, further including receiving, by a processor, ultrasonic data from an ultrasonic probe, generating, by the processor, an image quality score based on the ultrasonic data, and generating, by the processor, an ultrasonic examination score based on the image quality score.

[0026] The 20th embodiment is the method of the 11th embodiment, wherein the second subset includes any one of the ultrasonic images of at least one second anatomical structure, at least one ultrasonic image generated in the second imaging mode, or at least one ultrasonic image generated according to the second inspection protocol.

[0027] The accompanying drawings illustrate examples and are therefore exemplary embodiments and are not to be considered as limiting the scope.

Brief Description of the Drawings

[0028]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

[0029] A system and method for automated ultrasonic qualification are described. In some embodiments, a qualification system for issuing ultrasonic inspector qualifications to ultrasonic inspection candidates includes a computing device and an ultrasonic probe connected to the computing device and configured to generate ultrasonic data. The computing device is configured to generate an ultrasonic inspection score as part of an automated review based on the ultrasonic data. The computing device is configured to transfer an ultrasonic inspection candidate from an automated review to a manual review by an examiner based on the ultrasonic inspection score.

[0030] Conventional ultrasonic qualification systems can cause delays in the qualification process, burden the resources of a care facility, and there is a possibility that an ultrasonic operator may be qualified despite inadequate training. As a result, patients in need of ultrasonic examinations may receive only inferior care compared to the best available care.

[0031] Embodiments of the systems, devices, and methods for ultrasonic qualification disclosed herein constitute many advantages over conventional qualification systems. The embodiments disclosed herein eliminate bias and improve the speed of qualification compared to conventional ultrasonic qualification systems. The embodiments disclosed herein facilitate real-time feedback to candidates during ultrasonic examinations, enabling candidates to immediately incorporate non-delayed feedback into the ultrasonic examination while the examination remains fresh in the candidates' memory. This immediacy is not possible with conventional qualification systems that rely on feedback from manual reviewers. The embodiments of ultrasonic qualification disclosed herein are objective and unbiased in the consideration and grading of ultrasonic data submitted by qualification candidates. In contrast, conventional qualification systems can necessarily be subjective and biased because they rely solely on manual reviews. The embodiments disclosed herein facilitate a qualification process that can be standardized within and across different care facilities. In contrast, conventional qualification systems are typically ad hoc and not standardized across care facilities. The embodiments disclosed herein reduce the burden on resources within a care facility, such as the time requirements of reviewers (e.g., trained clinicians), compared to conventional qualification systems.

[0032] References to "one embodiment," "an embodiment," "one example," or "an example" in this specification mean that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment. The appearances of the phrases "in one embodiment" or "in an embodiment" in various places in this specification are not necessarily all referring to the same embodiment. The processes shown in the following figures are executed by processing logic comprising hardware (e.g., circuits, dedicated logic, etc.), software, or a combination of both. The processes are described below with respect to several sequential operations, but it should be understood that some of the operations described may be executed in a different order. Further, some operations may be executed in parallel rather than sequentially.

[0033] As used herein, the term "and / or" represents three relationships between possible objects. For example, A and / or B can represent the case where only A exists, the case where both A and B exist, and the case where only B exists, and A and B may be singular or plural.

[0034] FIG. 1 is a diagram 100 showing an environment for implementing a qualification system 101 according to some embodiments. The qualification system includes an ultrasonic system that can be operated by a candidate 105 (e.g., a student being trained to become a qualified ultrasonic operator). The ultrasonic system 101 includes a probe 103 and a computing device 102 connected to the probe via a communication link. The communication link between the probe and the computing device can be wired, wireless, or a combination thereof. The computing device 102 can include any suitable computing device such as a tablet, smartphone, head-up display, goggles, glasses, ultrasonic machine, or a combination thereof. As shown in FIG. 1, the computing device 102 can include one or more neural networks (NNs) 106 that can be at least partially implemented in the hardware of the computing device. The computing device 102 also includes a candidate qualification application 107 that can be implemented by a processor of the computing device (not shown in FIG. 1). For example, the memory of the computing device can store instructions that, when executed by the processor, cause the computing device to implement the candidate qualification application. As shown in FIG. 1, the candidate qualification application 107 is connected to a simulator system 108. In some embodiments, the simulator system 108 includes a dummy patient, and the ultrasonic probe transmits ultrasonic waves to the dummy patient and generates ultrasonic data based on the reflection of the ultrasonic waves from the dummy patient, as will be described in more detail below.

[0035] As shown in FIG. 1, an ultrasonic system 101 is connected to a network 109 (e.g., a computing network). In some embodiments, one or more of the probe 103 and the computing device 102 are connected to the network 109 to connect the ultrasonic system to the network. In one example, the network includes a care facility network, such as a secure Wi-Fi network operated by a care facility (e.g., a hospital). The qualification system can include a server system 111 connected to the network 109, whereby the server system communicates with the computing device 102 operated by a candidate. The server system 111 can include a qualification database for archiving data related to the qualification of the candidate, such as test results, reviewer feedback, ultrasonic data, metadata related to the ultrasonic data, etc. In one example, the server system includes a database of neural networks, and the computing device 102 operated by the candidate can include one or more neural networks supplied by the server system. For example, the neural network 106 can include one or more neural networks received from the server system 111. In some embodiments, the qualification application 107 determines the type of test being performed by the candidate, and the server system searches for a neural network suitable for the test type and provides the neural network to the qualification application for implementation on the candidate's computing device. In some embodiments, the candidate's computing device includes a plurality of computing devices. In some embodiments, the candidate's computing device includes the server system and a neural network implemented "in the cloud" by the server. Alternatively, the server system can be separated from the candidate's computing device, such as a server system maintained by a care facility.

[0036] In addition, the qualification system can also include a reviewer computing device 113 operated by a reviewer 112, such as a trained and qualified ultrasonic examiner, to manually review and grade ultrasonic data submitted by candidate 105 as part of the qualification process for the candidate. The reviewer computing device 113 can include a reviewer qualification application 114 that can be implemented by a processor of the reviewer computing device (not shown in FIG. 1). For example, the memory of the reviewer computing device can store instructions that, when executed by the processor, cause the reviewer computing device to implement the reviewer qualification application. The reviewer can perform any suitable function as part of the qualification review, such as reviewing the ultrasonic data supplied by the candidate via the user interface of the reviewer qualification application, and providing feedback (e.g., comments, grading, etc.) to the candidate and archiving the data on the server system.

[0037] During the ultrasonic examination of patient 104, ultrasonic examination candidate 105 can use an ultrasonic system 101 (e.g., probe 103 and computing device 102) to generate ultrasonic data for qualification by the qualification system. In one example, unlike conventional qualification systems, the qualification system does not necessarily send the ultrasonic data to a reviewer for manual review. Instead, it implements one or more neural networks 106, such as on candidate computing device 102, to automatically determine the characteristics of the ultrasonic data and provide characteristic-based feedback to candidate 105 in real time during the ultrasonic examination. The feedback generated by the neural network can also be communicated by computing device 102 to server system 111 for archiving as part of the qualification.

[0038] Neural network 106 can generate any suitable output based on ultrasonic data. In some embodiments, the qualification system implements a neural network to generate a quality metric for an ultrasonic image, such as a number between 0 and 1, where 0 indicates low quality and 1 indicates excellent quality, or a binary label such as "pass" or "fail". In some embodiments, the binary label is generated by applying a threshold to the probability of an indicator generated by the neural network, such as the probability that the ultrasonic image includes an acceptable view. In some embodiments, the threshold is an image quality threshold or other indicator threshold. In some embodiments, the image quality metric indicates that the image has sufficient quality to perform a given operation. In some embodiments, the neural network is trained to directly generate a binary label by an appropriate selection of a loss function used to train the neural network (e.g., first generating a probability without applying a threshold to it). Candidate qualification application 107 can display the quality metric to the candidate via the user interface of computing device 102. In addition to or instead of this, candidate qualification application 107 can generate a grade based on the quality of the ultrasonic image, such as by assigning a letter grade A, B, C, D, or F based on the quality metric generated by neural network 106. The grade can indicate the usefulness of the ultrasonic data, such as whether the ultrasonic image generated from the ultrasonic data is "good enough" to determine a pneumothorax condition.

[0039] Neural network 106 can receive any suitable input and generate a quality metric and / or grade for the candidate. As described above, the neural network can receive one or more ultrasonic images generated during an ultrasonic examination by candidate 105. Additionally, the neural network can include one or more secondary (e.g., additional) inputs for generating a quality metric and / or grade for the candidate, as will be described in more detail with respect to FIG. 3.

[0040] In some embodiments, the ultrasonic probe 103 includes an inertial measurement unit (IMU) that can measure one or more of force, acceleration, angular velocity, and magnetic field. The IMU can include a combination of an accelerometer, a gyroscope, and a magnetometer, and can generate position and / or orientation data including data representing six degrees of freedom such as yaw angle, pitch angle, and roll angle in a coordinate system. In addition to or instead of this, the ultrasonic system can include a camera for determining the position and / or orientation data of the ultrasonic probe. The position and / or orientation data can indicate the movement of the probe. The neural network can process the position and / or orientation data as a secondary input in addition to the ultrasonic image.

[0041] In some embodiments, the neural network also receives, as a secondary input, a display of the behavior of the candidate being examined. For example, an inexperienced or untrained ultrasonic operator may move the probe more than an experienced or trained operator during an ultrasonic examination, which can cause discomfort to the patient. Thus, an example of the secondary input indicating the behavior of the candidate includes the movement data of the candidate. For example, the candidate can wear a motion sensor on their clothing that determines the movement data of the candidate being examined (e.g., the degree of movement of the candidate's arm). In another example, the probe includes one or more motion sensors (e.g., IMU, gyro, position / orientation sensor, movement sensor, or other motion sensor) for collecting the movement data of the candidate. In addition to or instead of this, a camera in the examination room can generate the movement data. In one example, the secondary input to the neural network includes the amount of time, such as the time from when the candidate starts acquiring an ultrasonic image using the probe / ultrasonic system until the candidate generates and saves the ultrasonic image for review. For example, an "inexperienced" operator may take a long time to move the probe to the appropriate location to obtain an appropriate ultrasonic image, and this excessive time can have an adverse effect on the patient.

[0042] In addition to or instead of this, the secondary input indicative of the candidate's behavior can include audio content, including audio content spoken by the candidate, the patient, and / or another person watching the examination. For example, the audio content can include a conversation between the candidate and the patient, whereby the neural network can be trained to generate a better score for the candidate when positive communication is taking place with the patient, such as by telling the patient what to expect next during the examination. In one example, the patient says "You are hurting me" or someone in the room tells the candidate to move the probe in a particular way (e.g., the candidate gets assistance to get into the proper field of view). FIG. 2 is a diagram 200 showing an ultrasonic probe 201 having a sensor area 205 for determining the orientation of a grip according to some embodiments. In some embodiments, the ultrasonic probe 201 corresponds to the ultrasonic probe 103 or another ultrasonic probe. In some embodiments, the ultrasonic probe 201 corresponds to an ultrasonic probe having a sensor area for determining the orientation of a grip as described in U.S. Patent Application No. 18 / 045,477, Attorney Docket No. 105767P025, entitled "CONFIGURING ULTRASOUND SYSTEMS BASED ON SCANNER GRIP", filed on October 11, 2022, which is hereby incorporated by reference in its entirety. As shown in FIG. 2, the ultrasonic probe 201 includes a sensor area 205 for detecting the orientation of the grip and a transducer connected to the lens 203. In FIG. 2, the sensor area 205 is shown as an ellipsoid. However, the sensor area 205 can be of any suitable shape. In some embodiments, the sensor area 205 substantially covers the surface of the ultrasonic probe 201, for example, the sensor area can substantially cover substantially all of the ultrasonic probe 201, including or excluding the lens 203. In some embodiments, the sensor area 205 covers the area that the user normally holds (e.g., the grip portion of the probe that is narrower than the transducer head portion of the probe).

[0043] In some embodiments, the transducer of the ultrasonic probe 201 includes an ultrasonic transducer array and electronic equipment connected to the ultrasonic transducer array, transmits ultrasonic signals to the anatomical structure of a patient, and receives ultrasonic signals reflected from the anatomical structure of the patient. The ultrasonic probe 201 can include any suitable type of sensor for determining the orientation of the grip, in, above, or below the sensor region 205. In some embodiments, the ultrasonic probe 201 includes a capacitance sensor that can measure capacitance or changes in capacitance caused by a user's touch or proximity to a touch, as is common in touch screen technology. In addition to or instead of this, the ultrasonic probe 201 can include a pressure sensor configured to determine the magnitude of pressure caused by a user gripping the probe.

[0044] In some embodiments, the ultrasonic system receives sensor data from sensors in the sensor region 205 and generates a grip map 207 representing the sensor data. As shown in FIG. 2, the grip map 207 includes sensor data in a two-dimensional (2D) grid (e.g., a matrix) format. The nodes of the grid can correspond to the sensors in the sensor region 205 and can include the sensor data from the sensors. As an example, each intersection of the cross-hatching shown in the sensor region 205 of FIG. 2 can correspond to a sensor for determining the orientation of the grip and thus to a node in the 2D grid. The sensor data can include a multi-level indicator indicating the magnitude of the pressure on the sensor, such as on an integer scale from 0 to 5. For example, "0" can indicate that no pressure from the user's hand is detected by the sensor, "1" can indicate that a small amount of pressure from the user's hand is detected by the sensor. "2" can indicate that a greater amount of pressure from the user's hand than "1" is detected by the sensor, and "5" can indicate that the maximum amount of pressure from the user's hand is detected by the sensor. A neural network (e.g., NN106 in FIG. 1) can receive the orientation of the grip, such as the grip map 207, as a secondary input in addition to the ultrasonic image to generate a candidate quality metric and / or grade. Other examples of secondary inputs can include pressure data (e.g., how much pressure is applied from the probe to the patient), time data (e.g., the time it takes for the candidate to operate the ultrasonic system to perform an ultrasonic examination), and the like.

[0045] FIG. 3 is a diagram 300 showing a system that generates data related to the evaluation of a candidate based on one or more ultrasonic images and one or more secondary inputs according to some embodiments. As shown in FIG. 3, neural network 303 receives ultrasonic image 301 as a primary input and one or more secondary inputs 305, as described above, in order to generate output 309. Neural network 303 is an example of neural network 106 of FIG. 1. In some embodiments, output 309 includes data related to the evaluation of the candidate, such as quality metrics, grades, and / or scores of the candidate during the ultrasound examination. Neural network 303 can combine ultrasonic image 301 with one or more secondary inputs 305 in any suitable manner. In some embodiments, the neural network concatenates the secondary input and the ultrasonic image and processes the concatenated data in the upper (first) layer of the neural network. In some embodiments, the neural network processes the ultrasonic image in one or more layers of the neural network and concatenates the result (e.g., a feature map based on the ultrasonic image) with the secondary input for subsequent layers of the neural network.

[0046] In some embodiments, neural network 303 includes a plurality of networks and / or sections. Each section can include one or more neural networks. In some embodiments, the neural network uses two sections to process an ultrasonic image, and the second section receives a secondary input. Neural network 303 can combine one or more of the results output from the first and second sections with one or more of the ultrasonic image and the secondary input. In some embodiments, one or more ultrasonic images 301 are input into a first neural network, and the output of the first neural network (e.g., a feature map, or other output) is combined with one or more secondary inputs 305 and input into a second neural network to generate an output 309. In some embodiments, one or more secondary inputs 305 are input into a first neural network, and the output of the first neural network is combined with one or more ultrasonic images 301 and input into a second neural network to generate an output 309.

[0047] In some embodiments, neural network 303 receives a weight vector that assigns weights relative to the secondary inputs 305. For example, the weight vector can include user-assigned values from 0 to 1 to place more or less emphasis on the secondary inputs, such as by weighting probe direction data more heavily than audio data. By enabling the weights to include weights assigned by the user, the qualification system can be adjusted to match current trends in ultrasonic inspections over time. For example, currently, state-of-the-art may place significant emphasis on temporal data (e.g., the time it takes a candidate to perform an ultrasonic inspection). In the future, the state of the art may change and place emphasis on pressure data. The qualification system can easily accommodate these changes via the weight vector. In some embodiments, the qualification system qualifies candidates based on a combination of an automated review by neural network 303 and a manual review performed by a reviewer (e.g., reviewer 112 of FIG. 1). For example, the qualification system can limit the review of ultrasonic data to an automated review of the ultrasonic data using the candidate qualification application and the neural network described above until a threshold of competence is met. Once the threshold of competence is met, the qualification system can provide the ultrasonic data from the candidate to the reviewer for review as part of a manual review.

[0048] As an example of a threshold of ability, a candidate may receive a passing score (e.g., a binary "pass" label or a letter grade such as A or B) for a specified number of tests, such as the most recent five tests. In addition to or instead of this, the threshold of ability may require passing a certain test (e.g., receiving an acceptable letter grade) for, such as a bladder scan or a scan by an ultrasound protocol. An example of an ultrasound protocol is the extended focused assessment with sonography for trauma (eFAST) designed to detect ascites, pericardial fluid, pneumothorax, and / or hemothorax in trauma patients. In one example, for a lower extremity vascular echo examination for diagnosing the presence or absence of deep vein thrombosis (DVT), the qualification system can evaluate whether several reference cross-sections are continuously observed from the central side to the peripheral side. In another example, for an abdominal examination, if the order of observing multiple representative sites is predefined for each hospital, the qualification system can evaluate whether the examination was performed in the predefined order. Thus, the qualification system can transfer a candidate from an automated review to a manual review based on the history of tests performed by the candidate.

[0049] In some embodiments, to qualify a candidate, the qualification system imposes several review iterations, each iteration including first an automated review by a neural network, followed by a manual review by a reviewer upon successful completion of the automated review. In some embodiments, the qualification system groups the review process iterations based on one or more characteristics, such as difficulty, content, patient availability, and other characteristics. For example, the first iteration can include an ultrasound examination of the bladder, the second iteration can include an ultrasound examination by an ultrasound protocol, the third iteration can include an ultrasound examination of the anatomical structure of the heart, the fourth iteration can include an ultrasound examination using a simulator system (e.g., simulator system 108) (to be described in more detail later), and so on.

[0050] In some embodiments, the qualification system includes a qualification database that stores the examination results and examination data of ultrasonic examination candidates. For example, the qualification database can store data used as inputs to a neural network, and data generated by the neural network as shown in FIG. 3, such as ultrasonic images, sensor data, movement data, direction data, voices spoken by the candidate and / or patient, grip maps, examination scores, image quality scores, etc. The server system 111 of FIG. 1 can include the qualification database. The data stored in the qualification database can be provided as inputs to the neural network. Thus, the neural network can generate an examination score for an ultrasonic examination candidate based not only on data obtained from the current examination performed by the candidate, but also on data obtained from past examinations performed by the candidate. The qualification system can be configured to weight previous examinations differently from current examinations, for example, by using a more difficult threshold (such as a higher score threshold) for the current examination and an easier threshold for previous examinations. In one example, the data from the qualification database is provided to a decision tree that deterministically generates an examination score based on the history of data of the ultrasonic examination candidate. In addition to or instead of this, the data from the qualification database can be provided to a neural network trained to generate an examination score from the history of data of the ultrasonic examination candidate.

[0051] FIG. 4 is a data flow diagram of a process 400 implemented by a qualification system for issuing an ultrasonic inspection qualification to an ultrasonic inspection candidate in some embodiments. Referring to FIGS. 1 and 4, the qualification system includes an ultrasonic system including a probe 103 connected to a computing device 102. In some embodiments, the computing device includes one or more processors and a memory connected to the processor for executing process 400. In some embodiments, process 400 is executed using processing logic. The processing logic can include hardware (circuits, dedicated logic, etc.), software (such as that executed on a general-purpose computer system or a dedicated machine), firmware, or combinations thereof. In block 401, ultrasonic data is generated using an ultrasonic probe. In block 402, based on the ultrasonic data, an ultrasonic inspection score is generated as part of an automated review. In some embodiments, the computing device is configured to implement one or more neural networks 106 to generate an image quality score based on the ultrasonic data as part of the automated review, and the ultrasonic inspection score is based on the image quality score. In some embodiments, the computing device is implemented to determine an anatomical structure based on the ultrasonic data. The computing device can then select a neural network based on the anatomical structure from among a plurality of neural networks.

[0052] In some embodiments, the ultrasonic probe includes a touch sensing surface, and the processor is implemented to generate a grip orientation on the touch sensing surface. As described above, the processor can represent the grip orientation as a grip map. The computing device is implemented to generate an ultrasonic examination score based on the grip orientation. In some embodiments, the ultrasonic probe includes a pressure sensor implemented to generate pressure data indicating the magnitude of the pressure of the ultrasonic probe on the patient, and the computing device is implemented to generate an ultrasonic examination score based on the pressure data. In some embodiments, the qualification system includes an audio processor implemented to record audio content, and the computing device is implemented to generate an ultrasonic examination score based on the audio content.

[0053] In some embodiments, the ultrasound probe includes an inertial measurement unit implemented to generate movement data of the ultrasound probe, and the computing device is implemented to generate an ultrasound examination score based on the movement data. In some embodiments, the qualification system includes a sensor system implemented to generate movement data of an ultrasound examination candidate (e.g., data indicating how the ultrasound examination candidate moves during the examination), and the computing device is implemented to generate an ultrasound examination score based on the movement data. In some embodiments, the sensor system includes a wearable sensor configured to be worn by the ultrasound examination candidate. The movement data can be generated based on data sensed by the wearable sensor. In some embodiments, the qualification system includes a simulator system (e.g., simulator system 108 of FIG. 1). The simulator system can include a dummy patient, and the ultrasound probe can be implemented to transmit ultrasound to the dummy patient and generate ultrasound data based on the reflection of the ultrasound from the dummy patient. As shown in FIG. 4, at block 403, the computing device transfers data related to the ultrasound examination candidate from automated review to manual review by a reviewer based on the ultrasound examination score. By transferring the ultrasound examination candidate from neural network-based automated review to reviewer-based manual review, the computing device improves the ultrasound examination candidate to approach the goal of issuing an ultrasound examination certificate through the qualification process. In some embodiments, the qualification system anonymizes the candidate's data when the candidate is transferred from automated review to manual review, such that the reviewer cannot identify the candidate. This anonymization can remove the reviewer's inherent bias.

[0054] In some embodiments, as described above, the qualification system includes a qualification database that stores the examination results of ultrasonic examination candidates. The computing device can transfer an ultrasonic examination candidate from an automatic review to a manual review based on the examination results from the qualification database for at least some of the ultrasonic examination candidates different from the current ultrasonic examination candidate. In some embodiments, the qualification system is trained to look for what they have in common, such as ultrasonic examination candidates who have passed a manual review and what types of examinations they have passed in an automatic review. Then, the qualification system can transfer the data related to the current ultrasonic examination candidate from an automatic review to a manual review only if the current ultrasonic examination candidate has also passed the examinations in the automatic review that other ultrasonic examination candidates have passed. In this way, the qualification system can use the data of other ultrasonic examination candidates as a predictor of the ability of the current ultrasonic examination candidate. The data from the qualification database can be provided to a neural network trained to predict the ability of the current ultrasonic examination candidate. The neural network may be a neural network as shown in FIG. 1 or an additional neural network.

[0055] In some embodiments, the qualification system uses data of multiple ultrasonic examination candidates (e.g., data from the qualification database) to determine examination scores such as the grade of the current candidate. For example, the qualification system can grade "on a curve" by looking at the multiple test scores of multiple candidates and assigning a passing grade to the upper half of the candidates and a failing grade to the lower half of the candidates according to those test scores, or by assigning a passing grade to the top 30% of the candidates and a failing grade to the bottom 70% of the candidates.

[0056] FIG. 5 is a data flow diagram of a process 500 implemented by a qualification system for issuing ultrasonic inspection qualifications to ultrasonic inspection candidates in some embodiments. Referring to FIGS. 1 and 5, the qualification system includes an ultrasonic system 101 configured to generate ultrasonic images (block 501). The qualification system includes a candidate qualification application 107 that is at least partially implemented in the hardware of the qualification system and is configured to generate an image quality score for a first subset of ultrasonic images (block 502) and communicate a second subset of ultrasonic images to a reviewer computing device based on the image quality score (block 503). For example, the first subset of images can correspond to a first anatomical structure being imaged (e.g., the bladder), a first ultrasonic protocol (e.g., eFAST), a first examination preset (e.g., the bladder preset), etc., and the second subset of images can correspond to a second anatomical structure being imaged (e.g., the lung), a second protocol (e.g., a protocol other than eFAST), a second examination type (e.g., the cardiac preset), etc.

[0057] In some embodiments, the first subset of ultrasonic images and the second subset of ultrasonic images are disjoint from each other. In some embodiments, the qualification system includes a reviewer computing device 113 connected to a computing device 102. In some embodiments, the candidate qualification application determines guidance for an ultrasonic inspection candidate to improve at least one of the image quality scores based on at least one of the image quality scores falling below a threshold score. The qualification system can include a display device that displays the guidance in a visual representation. In some embodiments, the visual representation includes at least one of a training video, an icon of an ultrasonic probe, an arrow indicating a direction to move the ultrasonic probe, and an icon of the orientation of a grip for holding the ultrasonic probe.

[0058] In some embodiments, the candidate qualification determination application communicates a second subset of ultrasound images to the reviewer computing device based on at least one of the image quality score ratios exceeding a threshold score. For example, the candidate qualification determination application communicates a second subset of ultrasound images when the ratio of the image quality score is greater than the threshold score. In some embodiments, the first subset includes at least one ultrasound image of a particular anatomical structure. In some embodiments, the first subset includes at least one ultrasound image generated using the ultrasound system in a specified imaging mode. In some embodiments, the first subset includes at least one ultrasound image generated according to a specified examination protocol.

[0059] In some embodiments, the candidate qualification determination application includes a neural network that generates an image quality score. In some embodiments, the neural network generates the image quality score based on at least one of the orientation of the grip of the ultrasound probe of the ultrasound system (e.g., the grip map as described above with respect to FIG. 2), the amount of movement of the ultrasound probe, the amount of movement of the ultrasound examination candidate, the audio content, the magnitude of the pressure applied to the patient by the ultrasound probe, and the amount of time taken by the ultrasound examination candidate to operate the ultrasound system to generate the ultrasound image.

[0060] FIG. 6 is a data flow diagram of a process 600 implemented by an ultrasonic inspection qualification system according to some embodiments. Referring to FIGS. 1 and 6, the ultrasonic inspection qualification system includes an ultrasonic probe 103 configured to generate ultrasonic data (block 601). The ultrasonic inspection qualification system includes a computing device 102 configured to generate an ultrasonic image based on the ultrasonic data (block 602). The ultrasonic inspection qualification system includes a neural network 106 at least partially implemented in the hardware of the computing device to generate an image quality score based on the ultrasonic image (block 603). The ultrasonic inspection qualification system includes a qualification device (e.g., at least partially implemented by a processor of the computing device) configured to issue an ultrasonic inspector qualification based on the image quality score (block 604).

[0061] In some embodiments, the ultrasonic inspection qualification system issues a certificate based on the history of the candidate's scores. In some embodiments, the ultrasonic inspection qualification system issues a certificate if the candidate's score is greater than two-thirds of the scores of other candidates. In some embodiments, the ultrasonic inspection qualification system issues a certificate when the candidate is in the top one-third of all candidates or meets other criteria.

[0062] In some embodiments, the qualification system provides guidance to the candidate during the ultrasound examination. For example, the guidance can include hints displayed by a candidate qualification application on a user interface of a computing device. The qualification system can use one or more of the neural networks to determine the guidance. For example, the neural network can determine, based on the ultrasound data (e.g., ultrasound image), that it is necessary to adjust the imaging parameters to improve the quality of the ultrasound image. Examples of imaging parameters include gain, depth, and type. The neural network can generate an adjustment of the imaging parameters, and the candidate qualification application can display, on the user interface, a message for adjusting the imaging parameters according to the recommended adjustment from the neural network.

[0063] In some embodiments, the guidance includes a display of the movement of the probe. The qualification system can communicate the guidance via the user interface, for example, by displaying a direction arrow, broadcasting an audible recommendation such as "move the probe towards the center of the patient", providing tactile feedback of the probe, combinations thereof, and the like. In some embodiments, the guidance includes a recommendation for specific training for the candidate. For example, the qualification system can determine training materials from a database of training materials based on one or more of the neural network output, the type of ultrasound examination being performed, the anatomical structure being imaged, the imaging parameters, and the ultrasound data, to improve the candidate's ultrasound examination technique. The server system can maintain a database of training materials and provide recommended training materials to the candidate qualification application in response to a request from the candidate computing device.

[0064] In some embodiments, the qualification system generates guidance based on requests (e.g., candidate requests) provided by a candidate to the qualification system and communicates the guidance to the candidate computing system. The candidate requests can be spoken, typed, gestured, etc. For example, a candidate can speak "Help, how do I hold the probe?" or gesture using the probe in a specified way, such as swiping an "X" in the air to indicate that help is needed. In some embodiments, the qualification system can generate guidance for a candidate without an explicit request from the candidate. For example, the qualification system can determine from the output of a neural network within a computing device that the candidate has not passed an ultrasound examination (e.g., the neural network is generating a quality metric corresponding to a failing score). Accordingly, the qualification system can communicate to the candidate guidance instructing the candidate on the proper use of the ultrasound system. In some embodiments, when the qualification system communicates guidance to a candidate for current ultrasound data, the qualification system does not accept the current ultrasound data for qualifying the candidate. Rather, the qualification system is required to generate new ultrasound data without the guidance provided to the candidate for the new ultrasound data and submit this new ultrasound data for consideration towards qualification.

[0065] Referring back to FIG. 1, the qualification system includes a simulator system 108 connected to a computing device operated by candidate 105. The simulator system 108 includes hardware, software, firmware, or a combination thereof for emulating an ultrasonic examination of a living patient when a living patient is not available. For example, in some types of ultrasonic examinations that require emergency treatment such as detection of free fluid and the patient's condition, since such conditions may endanger life, it may not be practically possible for a candidate to approach a living patient in such a condition and perform an ultrasonic examination during training for qualification purposes. Therefore, the qualification system includes a simulator system capable of generating image data. For example, the qualification system can include a dummy patient (e.g., torso and head, full-body dummy, etc.) that can be imaged by an ultrasonic probe. The dummy patient can include artificial anatomical structures, and the ultrasonic probe and the computing device can generate ultrasonic images of the artificial anatomical structures.

[0066] In some embodiments, the simulator system generates image data that mimics an ultrasonic image but is not directly derived from the ultrasonic signals transmitted by the ultrasonic system. For example, the image data can be generated by the simulator system based on probe position and orientation data such as contact points or contact areas on the dummy patient, and data corresponding to the six degrees of freedom of the probe. In some embodiments, the simulator system generates image data based on imaging parameters set by the candidate. The position data, orientation data, and imaging parameters can be aggregated into vectors, for example, and supplied as inputs to the neural network 106. The neural network can be trained to generate an image that looks like an ultrasonic image, and this image can be used by the qualification system to authenticate the candidate.

[0067] In some embodiments, the images generated by the simulator system 108 are reviewed by one or more neural networks of the candidate's computing device 102 as part of the automated review, as described above. For example, the neural network can generate an image quality metric or an image grade. In some embodiments, the qualification system determines the image quality metric or grade based on the ground truth image. For example, the qualification system can include a database of ground truth images collected by experts trained on various anatomical structures and examination types. The qualification system can compare the image generated by the simulator system with a ground truth image, such as mean squared error (e.g., pixels or features extracted from the image), to determine the image quality metric or grade. In some embodiments, the neural network receives a ground truth image as a second (or conditional) input in addition to the image generated by the simulator system to generate an image quality metric or grade of the image generated by the simulator system.

[0068] FIG. 7 is a data flow diagram of a method 700 implemented by a computing device according to some embodiments, such as the computing device 102 of FIG. 1. Method 700 includes, at block 701, receiving ultrasonic data from an ultrasonic probe connected to the computing device. In some embodiments, the ultrasonic probe transmits an ultrasonic signal to a dummy patient and is instructed by the computing device to determine ultrasonic data based on the ultrasonic signal. At block 702, an image quality score is generated by the computing device based on the ultrasonic data. In some embodiments, the image quality score is generated using a neural network at least partially implemented in the hardware of the computing device. In some embodiments, an anatomical structure is determined based on the ultrasonic data, and the neural network is selected based on the anatomical structure from among a plurality of neural networks. At block 703, the ultrasonic data is communicated to a reviewer computing device for user authentication based on the image quality score.

[0069] In some embodiments, additional ultrasonic data is received from the ultrasonic probe, and an additional image quality score is generated by the computing device based on the additional ultrasonic data. In some embodiments, the computing device determines not to communicate the additional ultrasonic data to the reviewer computing device for user authentication based on the additional image quality score. In some embodiments, guidance for improving the additional image quality score is displayed on the user interface of the computing device. In some embodiments, the guidance includes at least one of instructions to move the probe, adjustment of imaging parameters, and type of examination. In some embodiments, training material is selected based on the additional image quality score and the additional ultrasonic data. In some embodiments, the training material is made available on the computing device for the user to utilize.

[0070] FIG. 8 is a data flow diagram of a method 800 implemented by a computing device according to some embodiments, such as the computing device 102 of FIG. 1. The method 800 includes, at block 801, receiving ultrasonic data from an ultrasonic probe connected to the computing device. At block 802, an image quality score is generated by the computing device based on the ultrasonic data. At block 803, based on the image quality score, the communication of the ultrasonic data to a reviewer computing device for user authentication is stopped. Since the image quality score is below a threshold quality score, the computing device can stop the communication of the ultrasonic data. Thus, the authentication system can determine that the ultrasonic data is not "good enough", and the reviewer spends time evaluating the ultrasonic data as part of a manual review. Thus, the authentication system is more efficient than a conventional authentication system that relies only on manual review and can waste the reviewer's time.

[0071] FIG. 9 is a data flow diagram of a method 900 implemented by a computing device according to some embodiments, such as the computing device 102 of FIG. 1. The method 900 includes, at block 901, receiving ultrasonic data from an ultrasonic probe connected to the computing device. At block 902, an image quality score is generated by the computing device based on the ultrasonic data. At block 903, the image quality score and the ultrasonic data are communicated to an authentication server for user authentication.

[0072] FIG. 10 is a data flow diagram of a method 1000 implemented by a computing device according to some embodiments, such as the computing device 102 of FIG. 1. The method 1000 includes receiving an ultrasonic image at block 1001. At block 1002, an image quality score for a first subset of the ultrasonic image is generated. At block 1003, a second subset of the ultrasonic image is communicated to a reviewer computing device for user authentication based on the image quality score. In some embodiments, the first subset of the ultrasonic image and the second subset of the ultrasonic image are different from each other. For example, the first subset of the image can correspond to a first anatomical structure being imaged (e.g., the bladder), a first ultrasonic protocol (e.g., eFAST), a first examination preset (e.g., a bladder preset), etc., and the second subset of the image can correspond to a second anatomical structure being imaged (e.g., the lung), a second protocol (e.g., a protocol other than eFAST), a second examination type (e.g., a heart preset), etc.

[0073] In some embodiments, the method 1000 includes communicating the first subset of the ultrasonic image and the image quality score to an authentication server for user authentication. In some embodiments, the method 1000 includes communicating a comment to a candidate computing device that generated the ultrasonic image. In some embodiments, the comment indicates transmitting the second subset to a reviewer computing device. In some embodiments, the comment includes at least one of the image quality scores of the first subset. In some embodiments, the comment indicates that at least one ultrasonic image of the first subset has an image quality score corresponding to a failing grade. In some embodiments, the method 1000 includes obtaining a number of image thresholds. In some embodiments, the comment indicates that the number of image thresholds of at least the ultrasonic images of the first subset has an image quality score corresponding to a passing grade.

[0074] FIG. 11 is a data flow diagram of a method 1100 implemented by a computing device according to some embodiments such as the computing device 102 and / or the reviewer computing device 113 of FIG. 1. The method 1100 includes receiving an ultrasonic image at block 1101. At block 1102, an image quality score in the ultrasonic image is generated using a neural network at least partially implemented in the hardware of the computing device. At block 1103, an ultrasonic inspection qualification is issued based on the image quality score.

[0075] FIG. 12 is a block diagram of an exemplary computing device 1200 capable of performing one or more of the operations described herein according to some embodiments. The computing device 1200 may be connected to other computing devices within a LAN, intranet, extranet, and / or the Internet. The computing device may operate in the capacity of a server machine in a client-server network environment or in the capacity of a client in a peer-to-peer network environment. The computing device may be provided by any machine capable of executing a set of instructions (sequential or otherwise) that specify the actions to be taken by that machine, such as a personal computer (PC), server computing, desktop computer, laptop computer, tablet computer, smartphone, or the like. Further, although only a single computing device is shown, the term "computing device" should also be construed to include any collection of computing devices that individually or together execute a set (or sets) of instructions to perform the methods described herein. In some embodiments, the computing device 1200 can be one or more of an access point and a packet transfer component.

[0076] Exemplary computing device 1200 can include a processing device (e.g., a general-purpose processor, PLD, etc.) 1202, a main memory 1204 (e.g., synchronous dynamic random access memory (DRAM), read-only memory (ROM)), and a static memory 1206 (e.g., flash memory and data storage device 1218), which can communicate with each other via bus 1230. Processing device 1202 may be provided by one or more general-purpose processing devices such as a microprocessor, a central processing device, etc. In an exemplary example, processing device 1202 may include a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing another instruction set or a combination of instruction sets. Processing device 1202 may also include one or more dedicated processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. Processing device 1202 can be configured to perform the operations described herein in accordance with one or more aspects of the present disclosure in order to perform the operations and processes described herein.

[0077] Computing device 1200 can further include a network interface device 1208 that can communicate with network 1220. Computing device 1200 can also include a video display unit 1210 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 1212 (e.g., a keyboard), a cursor control device 1214 (e.g., a mouse), and an acoustic signal generating device 1216 (e.g., a speaker and / or a microphone). In one embodiment, video display unit 1210, alphanumeric input device 1212, and cursor control device 1214 may be combined into a single component or device (e.g., an LCD touch screen).

[0078] The data storage device 1218 can include a computer-readable storage medium 1228, and the computer-readable storage medium can store one or more sets of instructions 1226, such as instructions for performing the operations described herein, according to one or more aspects of the present disclosure. Also, the instructions 1226 may be wholly or at least partially present within the main memory 1204 and / or within the processing device 1202 during execution by the computing device 1200, the main memory 1204, and the processing device 1202 that also constitutes the computer-readable medium. Further, the instructions can be transmitted or received via the network 1220 via the network interface device 1208.

[0079] Although the computer-readable storage medium 1228 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" should be construed to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated cache and server) that store one or more sets of instructions. The term "computer-readable storage medium" should also be construed to include any medium that can store, encode, or carry a set of instructions for machine execution and cause the machine to perform the methods described herein. Thus, the term "computer-readable storage medium" should be construed to include, but not be limited to, solid-state memory, optical media, and magnetic media.

[0080] Unless otherwise specified, terms such as "transmit", "determine (judge)", "receive", "generate", etc. refer to operations and processes executed or performed by a computing device that manipulates data represented as physical (electronic) quantities in the registers and memories of the computing device and converts it into other data similarly represented as physical quantities in the memory or registers of the computing device or other such information storage devices, transmission devices, or display devices. Also, terms such as "first", "second", "third", "fourth", etc. used in this specification mean labels for distinguishing between different elements and do not necessarily imply an order according to their numerical designations.

[0081] Also, the examples described in this specification relate to an apparatus for performing the operations described in this specification. This apparatus may be specially configured for the required purpose or may comprise a general-purpose computing device selectively programmed by a computer program stored in a computing device. Such a computer program may be stored in a computer-readable non-transitory storage medium.

[0082] The methods and exemplary examples described in this specification are not inherently related to any particular computer or other device. It can be appreciated that various general-purpose systems can be used in accordance with the teachings described in this specification, or it may be convenient to construct more specialized devices for performing the required method steps. The structures required for various of these systems will appear as described in the above explanation.

[0083] The above description is illustrative and not restrictive. Although the present disclosure has been described with reference to specific exemplary embodiments, it is recognized that the present disclosure is not limited to the described embodiments. The scope of the present disclosure should be determined with reference to the following claims, along with the full scope of equivalents to which the claims are entitled.

[0084] As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. The terms "comprises", "comprising", "includes" and / or "including", when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Accordingly, the terms used herein are for the purpose of describing particular embodiments only and are not intended to be limiting.

[0085] It should also be noted that in some alternative implementations, the described functions / operations may occur out of the order described in the figures. For example, two figures shown in succession may in fact be executed substantially simultaneously, depending on the functions / operations involved, or sometimes in the reverse order.

[0086] Although method operations are described in a particular order, other operations may be performed between the described operations, the described operations may be adjusted so that they occur at slightly different times, or the described operations may be distributed in a system that allows the occurrence of processing operations at various intervals related to the processing.

[0087] Various units, circuits, or other components may be described or claimed as "configured to" or "configurable to" perform one or more tasks. In such contexts, the phrases "configured to" or "configurable to" are used to imply structure by indicating that the unit / circuit / component includes a structure (e.g., a circuit) that performs one or more tasks during operation. Thus, a unit / circuit / component can be said to be configured to perform a task, or configurable to perform a task, even if the specified unit / circuit / component is not currently operating (e.g., is not on). A unit / circuit / component that is described with the language "configured to" or "configurable to" includes hardware, such as a circuit, and a memory that stores program instructions executable to perform operations. Specifying that a unit / circuit / component is "configured to" or "configurable to" perform one or more tasks is clearly intended not to invoke 35 U.S.C. § 112, ¶ 6 with respect to that unit / circuit / component. Further, "configured to" or "configurable to" can include a general structure (e.g., a general-purpose circuit) that is operated by software and / or firmware (e.g., an FPGA or general-purpose processor that executes software) to operate in a manner capable of performing the task in question. "Configured to" can also include adapting a manufacturing process (e.g., semiconductor manufacturing equipment) to manufacture a device (e.g., an integrated circuit) adapted to perform one or more tasks or operations. "Configurable to" is clearly not intended to apply to a blank medium, an unprogrammed processor or unprogrammed general-purpose computer, or an unprogrammed programmable logic device, programmable gate array, or other unprogrammed device, unless accompanied by a programmed medium that gives the device the ability to be configured to perform the disclosed function.

[0088] The above description has been presented for purposes of illustration and is described with reference to specific embodiments. However, the above exemplary description is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the embodiments and their practical application, thereby enabling others skilled in the art to best utilize the embodiments and various modifications suitable for the particular use contemplated. Accordingly, the embodiments are to be considered illustrative rather than restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope of the appended claims and the equivalents thereof.

Claims

1. In a qualification system for issuing ultrasonic examiner qualifications to ultrasonic examination candidates, an ultrasonic system configured to generate ultrasonic images, at least partially implemented in the hardware of the qualification system, generate an image quality score for a first subset of the ultrasonic images, a candidate qualification application configured to communicate a second subset of the ultrasonic images to a reviewer computing device based on the image quality score, A qualification system comprising:

2. The qualification system according to claim 1, further comprising the reviewer computing device.

3. The qualification system according to claim 1, wherein the first subset and the second subset are different from each other.

4. The candidate qualification application determines guidance for the ultrasonic examination candidate for improving at least one of the image quality scores based on at least one of the image quality scores being below a threshold score, and further comprises a display device for displaying the guidance in a visual representation. The qualification system according to claim 1.

5. The visual representation includes at least one of a training video, an icon of an ultrasonic probe, an arrow indicating a direction in which the ultrasonic probe is to be moved, and an icon of an orientation of a grip for holding the ultrasonic probe. The qualification system according to claim 4.

6. The candidate qualification application is implemented to communicate the second subset of the ultrasonic images to the reviewer computing device based on at least one percentage of the image quality scores exceeding a threshold score, and the first subset includes one or more of the ultrasonic images of at least one first anatomical structure, at least one ultrasonic image generated in a first imaging mode of the ultrasonic system, or at least one ultrasonic image generated according to a first examination protocol. The qualification system according to claim 1.

7. The qualification system according to claim 1, wherein the candidate qualification application implements a neural network for generating the image quality score.

8. The neural network is implemented to generate the image quality score based on at least one of the orientation of the grip of the ultrasonic probe of the ultrasonic system, the amount of movement of the ultrasonic probe, the amount of movement of the ultrasonic examination candidate, the voice content, the magnitude of the pressure applied to the patient by the ultrasonic probe, and the amount of time spent by the ultrasonic examination candidate to operate the ultrasonic system to generate the ultrasonic image. The qualification system according to claim 7.

9. The candidate qualification application Receives ultrasonic data from an ultrasonic probe, Generates the image quality score based on the ultrasonic data, Generates an ultrasonic examination score based on the image quality score The qualification system according to claim 1, which is configured as described above.

10. The second subset includes any one of one or more of the ultrasonic images of at least one second anatomical structure, at least one ultrasonic image generated in a second imaging mode, or at least one ultrasonic image generated according to a second examination protocol. The qualification system according to claim 1.

11. A computer-implemented method for issuing ultrasonic examiner qualification to an ultrasonic examination candidate, comprising: Generating an ultrasonic image by a processor, Generating an image quality score for a first subset of the ultrasonic images by the processor, Communicating a second subset of the ultrasonic images to a reviewer computing device by the processor based on the image quality score Including the method.

12. The method according to claim 11, further comprising communicating the image quality score to the reviewer computing device by the processor.

13. The method according to claim 11, wherein the first subset and the second subset are different from each other.

14. The method according to claim 11, further comprising determining guidance for the ultrasonic examination candidate to improve at least one of the image quality scores based on the fact that at least one of the image quality scores is below a threshold score, and displaying the guidance in a visual representation.

15. The visual representation of claim 14 includes at least one of a training video, an icon of an ultrasonic probe, an arrow indicating a direction in which the ultrasonic probe is moved, and an icon of an orientation of a grip for holding the ultrasonic probe.

16. The method of claim 11 further includes communicating, by the processor, the second subset of the ultrasonic images to the reviewer computing device based on at least one percentage of the image quality scores exceeding a threshold score, wherein the first subset includes one or more of ultrasonic images of at least one first anatomical structure, at least one ultrasonic image generated in a first imaging mode of the ultrasonic system, or any of at least one ultrasonic image generated according to a first examination protocol.

17. The method of claim 11, wherein the processor implements a neural network for generating the image quality score.

18. The neural network of claim 17 is implemented to generate the image quality score based on at least one of an orientation of a grip of an ultrasonic probe of the ultrasonic system, an amount of movement of the ultrasonic probe, an amount of movement of the ultrasonic examination candidate, audio content, a magnitude of pressure applied to a patient by the ultrasonic probe, and an amount of time spent by the ultrasonic examination candidate to operate the ultrasonic system to generate the ultrasonic image.

19. Receiving, by the processor, ultrasonic data from an ultrasonic probe, Generating, by the processor, the image quality score based on the ultrasonic data, Generating, by the processor, an ultrasonic examination score based on the image quality score The method of claim 11 further includes.

20. The method of claim 11, wherein the second subset includes any of ultrasonic images of at least one second anatomical structure, at least one ultrasonic image generated in a second imaging mode, or at least one ultrasonic image generated according to a second examination protocol.

Images