Image sequence processing method and apparatus, computing device, and storage medium

Abstract

An image sequence processing method, device, computing equipment and storage medium are provided. The method can include: obtaining a first image sequence, the first image sequence being captured based on a first capture target for a first human body; obtaining at least one additional image sequence based on the first image sequence, the at least one additional image sequence being for the first human body, and the at least one additional image sequence having a different sequence type from the first image sequence; and processing the first image sequence and the at least one additional image sequence based on the first capture target to obtain an image processing result related to the first capture target.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing, and in particular, to an image sequence processing method and device, a computing device, and a storage medium. BACKGROUND

[0002] Currently, doctors or other analysts often perform medical analysis or diagnosis by using medical image sequences scanned by medical scanning devices (such as nuclear magnetic resonance imaging scanners, magnetic resonance imaging scanners, computed tomography scanners, etc.), or by using results of intelligent analysis (for example, regions of interest, lesion regions, etc. circled by intelligence) to assist in the analysis. Therefore, in order to more effectively assist doctors or other analysts, it is desirable to obtain a scheme for more effectively processing image sequences to obtain recognition or lesion detection results.

[0003] In addition, there can be a situation that a certain image sequence of a patient is currently acquired by using a certain acquisition device (for example, CT), but diagnosis of the current condition can be more accurate and effective in combination with another medical image sequence (for example, an image sequence acquired by using MR). In this case, doctors often need to determine different sequence types from the patient's historical images according to personal experience, manually select and pull the image sequences, input instructions, to obtain further comprehensive analysis results, which consumes more of the doctor's diagnosis time and thus can reduce the diagnosis efficiency.

[0004] The methods described in this section can not necessarily be the methods that have been previously conceived or adopted. Unless otherwise indicated, it should not be assumed that any of the methods described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more methods should not be assumed to have been raised with respect to any prior art merely by virtue of their inclusion in this section. SUMMARY

[0005] According to an aspect of the present disclosure, an image sequence processing method is provided, including: acquiring a first image sequence, the first image sequence being acquired based on a first acquisition target for a first human body; acquiring at least one additional image sequence based on the first image sequence, the at least one additional image sequence being for the first human body, and the at least one additional image sequence having a different sequence type from the first image sequence; and processing the first image sequence and the at least one additional image sequence based on the first acquisition target to obtain an image processing result related to the first acquisition target.

[0006] According to another aspect of the present disclosure, there is provided an image sequence processing apparatus, comprising: a first image sequence obtaining unit configured to obtain a first image sequence, the first image sequence being captured based on a first capturing target with respect to a first human body; an additional image sequence obtaining unit configured to obtain at least one additional image sequence based on the first image sequence, the at least one additional image sequence being with respect to the first human body and having a different sequence type from the first image sequence; and an image sequence processing unit configured to process the first image sequence and the at least one additional image sequence based on the first capturing target to obtain an image processing result related to the first capturing target.

[0007] According to another aspect of the present disclosure, there is provided a computing device, comprising: a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement an image sequence processing method according to one or more embodiments of the present disclosure.

[0008] According to another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements an image sequence processing method according to one or more embodiments of the present disclosure.

[0009] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements an image sequence processing method according to one or more embodiments of the present disclosure.

[0010] These and other aspects of the present disclosure will become apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0011] In the following description of the example embodiments, in conjunction with the accompanying drawings, more details, features, and advantages of the present disclosure will be disclosed. In the drawings:

[0012] Figure 1 is a schematic diagram illustrating an example system in which various methods described herein can be implemented according to an example embodiment;

[0013] Figure 2 is a flowchart illustrating an image sequence processing method according to an example embodiment;

[0014] Figure 3 is a flowchart illustrating part of steps of an image sequence processing method according to another example embodiment;

[0015] Figure 4is a schematic block diagram illustrating an image sequence processing apparatus according to an exemplary embodiment;

[0016] Figure 5 is a block diagram illustrating an exemplary computer device to which the exemplary embodiments can be applied. DETAILED DESCRIPTION

[0017] In the present disclosure, the use of the terms "first", "second", etc. to describe various elements is not intended to denote a position relationship, a time sequence relationship, or an importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element can refer to the same instance of the element, and in some cases, based on the context of the description, they can also refer to different instances.

[0018] The terms used in the description of various described examples in the present disclosure are only for the purpose of describing the specific examples, and are not intended to be limiting. Unless the number of elements is specifically limited, the element can be one or more, if the number of elements is not specifically limited. As used herein, the term "plurality" means two or more, and the term "based on" should be interpreted as "at least partially based on". In addition, the terms "and / or" and "at least one of" encompass any one and all possible combinations of the listed items.

[0019] Exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0020] Figure 1 is a schematic diagram illustrating an example system 100 in which various methods described herein can be implemented according to exemplary embodiments.

[0021] Referring to Figure 1 The system 100 includes a client device 110, a server 120, and a network 130 communicatively coupling the client device 110 and the server 120.

[0022] The client device 110 includes a display 114 and a client application (APP) 112 that can be displayed via the display 114. The client application 112 can be an application program that needs to be downloaded and installed before running or a lite app as a lightweight application. In the case that the client application 112 is an application program that needs to be downloaded and installed before running, the client application 112 can be pre-installed on the client device 110 and activated. In the case that the client application 112 is a lite app, the user 102 can directly run the client application 112 on the client device 110 without installing the client application 112 by searching for the client application 112 in a host application (e.g., by the name of the client application 112, etc.) or scanning a graphic code (e.g., a bar code, a QR code, etc.) of the client application 112, etc. In some embodiments, the client device 110 can be any type of mobile computer device, including a mobile computer, a mobile phone, a wearable computer device (e.g., a smart watch, a head-mounted device, including smart glasses, etc.), or other types of mobile devices. In some embodiments, the client device 110 can alternatively be a stationary computer device, such as a desktop computer, a server computer, or other types of stationary computer devices. In some alternative embodiments, the client device 110 can also be or can include a medical image printing device.

[0023] The server 120 is typically a server deployed by an Internet Service Provider (ISP) or an Internet Content Provider (ICP). The server 120 can represent a single server, a cluster of multiple servers, a distributed system, or a cloud server providing underlying cloud services such as cloud database, cloud computing, cloud storage, cloud communication. It will be understood that, although Figure 1 Although the server 120 is shown in FIG. 1 communicating with only one client device 110, the server 120 can simultaneously provide background services for multiple client devices.

[0024] Examples of the network 130 include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), and / or a combination of communication networks such as the Internet. The network 130 can be a wired or wireless network. In some embodiments, data exchanged over the network 130 is processed using technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. In addition, all or some links can be encrypted using encryption technologies such as the Secure Sockets Layer (SSL), Transport Layer Security (TLS), a Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. In some embodiments, custom and / or proprietary data communication technologies and / or formats can also be used instead of or in addition to the foregoing.

[0025] The system 100 can also include an image acquisition device 140. In some embodiments, Figure 1 The image acquisition device 140 shown can be a medical scanning device, including but not limited to a scanning or imaging device used in a positron emission tomograph 10 (PET), a positron emission tomograph 10 with computerized tomograph 10 (PET / CT), a single photon emission computed tomograph 10 with computerized tomograph 10 (SPECT / CT), a computerized tomograph 10 (CT), a medical ultrasonograph 10, a nuclear magnetic resonance imaging (NMRI), a magnetic resonance imaging (MRI), a cardiac angiograph 10 (CA), a digital radiograph 10 (DR), etc. For example, the image acquisition device 140 can include a digital subtraction angiography scanner, a magnetic resonance angiography scanner, a tomographic angiography scanner, a positron emission tomography scanner, a positron emission computed tomography scanner, a single photon emission computed tomography scanner, a computerized tomography scanner, a medical ultrasonography device, a nuclear magnetic resonance imaging scanner, a magnetic resonance imaging scanner, a digital radiography scanner, etc. The image acquisition device 140 can be connected to a server (e.g., the server 120 in the system 100 or a separate server of the imaging system not shown in the figure) to enable processing of image data, including but not limited to conversion of scan data (e.g., into a medical image sequence), compression, pixel correction, three-dimensional reconstruction, etc. Figure 1 The image acquisition device 140 can be connected to the client device 110, e.g., through the network 130, or otherwise directly connected to the client device to communicate with the client device.

[0026] The image acquisition device 140 can be connected to the client device 110, e.g., through the network 130, or otherwise directly connected to the client device to communicate with the client device.

[0027] Optionally, the system can further include a smart computing device or a computing card 150. The image acquisition device 140 can include or be connected (e.g., removably connected) to such a computing card 150, etc. As one example, the computing card 150 can implement processing of image data, including but not limited to conversion, compression, pixel correction, reconstruction, etc. As another example, the computing card 150 can implement the image sequence integration method according to embodiments of the present disclosure.

[0028] The system can further include other parts not shown, such as a data storage. The data storage can be a database, a data repository, or other form of one or more devices for data storage, which can be a conventional database, and can include a cloud database, a distributed database, etc. For example, the direct image data formed by the image acquisition device 140 or the medical image sequence or three-dimensional image data obtained after image processing, etc. can be stored in the data storage for subsequent retrieval by the server 120 and the client device 110 from the data storage. As one example, the obtaining of at least one additional image sequence as described in embodiments of the present disclosure can include obtaining at least one additional image sequence from the data storage, although the present disclosure is not limited thereto. In addition, the image acquisition device 140 described above can also directly provide the server 120 or the client device 110, etc. with the direct image data or the medical image sequence or three-dimensional image data obtained after image processing, etc.

[0029] The user can use the client device 110 to view the acquired images or images, including preliminary image data or images obtained after analysis and processing, view analysis results, interact with the acquired images or analysis results, input acquisition instructions, configuration data, etc. The client device 110 can send configuration data, instructions or other information to the image acquisition device 140 to control the acquisition and data processing of the image acquisition device, etc.

[0030] For the purposes of embodiments of the present disclosure, Figure 1In the example, client application 112 can be an image sequence management application that provides various functions, such as storage management, indexing, sorting, and classification of acquired image sequences. Correspondingly, server 120 can be a server used in conjunction with the image sequence management application. Server 120 can provide image sequence management services to client application 112 running on client device 110 based on user requests or instructions generated according to embodiments of this disclosure. For example, it can manage image sequence storage in the cloud, store and classify image sequences according to specified indexes (including, but not limited to, sequence type, patient identifier, body part, acquisition target, acquisition stage, acquisition machine, whether lesions are detected, severity, etc.), and retrieve and provide image sequences to client devices according to specified indexes, etc. Alternatively, server 120 can also provide or allocate such service capabilities or storage space to client device 110, whereby client application 112 running on client device 110 provides corresponding image sequence management services based on user requests or instructions generated according to embodiments of this disclosure, etc. It is understood that the above is only one example, and this disclosure is not limited thereto.

[0031] Figure 2 This is a flowchart illustrating an image sequence processing method 200 according to an exemplary embodiment. Method 200 can be implemented on a client device (e.g., ...). Figure 1 The execution is performed at the client device 110 shown, that is, the execution entity of each step of method 200 can be... Figure 1 The client device 110 shown. In some embodiments, method 200 can be performed on a server (e.g., Figure 1 The method 200 is executed at server 120 (as shown in the figure). In some embodiments, the method 200 may be executed in combination by a client device (e.g., client device 110) and a server (e.g., server 120).

[0032] The steps of method 200 are described in detail below.

[0033] refer to Figure 2 In step 210, a first image sequence is acquired, which is acquired based on a first acquisition target and targeting a first human body.

[0034] At step 220, at least one additional image sequence is obtained based on the first image sequence. The at least one additional image sequence is for the first human body and has a different sequence type from the first image sequence.

[0035] At step 230, the first image sequence and the at least one additional image sequence are processed based on the first acquisition target to obtain an image processing result related to the first acquisition target.

[0036] By the above method, the additional sequence of the same person but of different sequence types can be automatically acquired for analysis and recognition, so as to realize more intelligent image sequence processing based on active pulling of other sequences and more accurate recognition effect, so as to more conveniently and effectively assist doctors in analysis and reduce the additional work of doctors to acquire other sequences by themselves.

[0037] The image processing result can include any human body part recognition result, lesion or sign recognition result, lesion detection result, etc. that can be obtained from the image sequence (e.g. medical image sequence or human body image sequence) and understood by those skilled in the art, and the processing of the image sequence can be any processing corresponding to the acquisition target and the image processing result that can be understood by those skilled in the art. For example, the image processing result can include human body segmentation data (e.g. segmentation between tissues, blood vessel topology, boundary of bones and other structures), region of interest (e.g. region or interval containing a lesion or suspected lesion), possible lesion type and lesion location, features, size, extent, etc. and it can be understood that the present disclosure is not limited thereto.

[0038] It can be understood that the execution subject here can be the client device 110. For example, the client device 110 acquires the first image sequence and the at least one additional image sequence and processes these image sequences based on the computing capability of the client device 110. Alternatively, the execution subject can be different client devices 110, for example, one device acquires the first image sequence (from itself, from another device or from cloud storage, etc.), another device performs analysis and calculation based on the first image sequence and acquires the additional image sequence, and then controls another device (or server, computing card, etc.) to perform processing. As another example, the execution subject here can be the server 120, or can be a computing card 150 included in or connected to the image acquisition device 140 or the client device 110, etc. The server 120 or the computing card 150 can be connected to the required terminal device in various ways (wired or wireless) to control the terminal device to perform corresponding processing. It can be understood that the present disclosure is not limited thereto.

[0039] In addition, it can be understood that "acquiring the first image sequence" does not mean that the first image sequence must be actively pulled or called, but can include various steps of making the first image sequence exist, including but not limited to receiving a push from an acquisition device or other device to obtain the first image sequence, or the first image sequence is originally stored locally, etc.

[0040] The additional sequences can be obtained based on various rules, and some non-limiting example embodiments will be given below.

[0041] According to some embodiments, obtaining the at least one additional image sequence based on the first image sequence can comprise: determining at least one additional sequence type based on a first sequence type set associated with the first acquisition target; and obtaining pre-stored at least one additional image sequence associated with the first human body based on the at least one additional sequence type. The acquisition target can include a general or broad acquisition target, such as an acquisition of a certain specific department (“from cardiology”), an acquisition for a certain human body part (“lung imaging”), etc. The acquisition target can also include a more refined acquisition target, such as an acquisition for a specific lesion, pathology or sign, such as but not limited to “lung patchy shadow”, “whether the patient has pneumonia”, etc.

[0042] According to some embodiments, the first acquisition target can be for a first lesion, and the first sequence type set comprises one or more sequence types constructed based on the first lesion. For example, there can be pre-constructed sequence types associated with each of one or more lesions for assisting in the identification. When the first acquisition target is associated with a first lesion of the one or more lesions, then another one or more sequence types can be obtained based on the pre-constructed or pre-stored sequence type set, and if corresponding image sequences have been stored or acquired for the patient, then the corresponding one or more image sequences can be obtained (e.g., “actively pulled”) for processing and identification. In this way, a more complete image sequence and more accurate image processing result can be obtained for a specific lesion type.

[0043] According to embodiments of the present disclosure, appropriate sequences can be selected according to the acquisition target or the condition. The “selection” can be understood as an intelligent selection and targeted identification based on the lesion, and thus different lesions can employ specific sequence combinations to assist in the identification. For example, there can be a pre-set reference sequence list that is better for each disease, such as indicating that lesion 1 can be identified by sequences a, b, lesion 2 by sequences b, c, etc., and when the detected lesion is lesion 1, sequences a, b are selected for further analysis and identification to obtain features related to the lesion.

[0044] According to some optional additional embodiments, the method 200 can further comprise, prior to determining the at least one additional sequence type, analyzing the first image sequence based on the first acquisition target to determine a lesion location for the first lesion. In such embodiments, the first lesion sequence type set related to the first lesion comprises a sequence type subset related to a different lesion location of the first lesion, respectively. In this way, different sequences can be selected according to the lesion location.

[0045] Additionally or alternatively, referring toFigure 3 Method 200 may further include step 300, which further includes sub-steps 310-330. At step 310, preliminary lesion detection processing is performed based on the first image sequence to obtain detection results. At step 320, an experience base is queried based on the detection results to obtain associated sequences as one or more additional sequences. For example, the experience base may include a set of diagnostic reference sequence types, such as ultrasound sequences, CT sequences, MR sequences, etc., for a specific lesion. It is understood that the above are merely examples, and this disclosure is not limited thereto.

[0046] At optional step 330, the acquired additional sequence can be combined with the first image sequence for further processing, such as further lesion detection processing. Combination may include simple combination (e.g., data merging) or some basic image processing operations, such as sequence alignment, cropping, etc. As an example, preliminary lesion detection processing may include lower accuracy or confidence requirements than the final lesion detection processing to increase computational efficiency. It is understood that the above method step 300 can be performed after step 220 of method 200 and before step 230, or can replace step 220.

[0047] According to some optional additional embodiments, method 200 may further include, before determining at least one additional sequence type: analyzing a first image sequence based on a first acquisition target to determine the lesion severity for a first lesion. In such embodiments, the set of first lesion sequence types associated with the first lesion includes subsets of sequence types respectively associated with different lesion severityes of the first lesion. Thus, different sequences can be selected according to the lesion severity. The lesion severity may include the size, color, shape, etc., of the lesion area or sign features, or may include several levels determined based on the lesion or sign in the image, which can be obtained through image analysis algorithms understood by those skilled in the art, and will not be elaborated here.

[0048] For example, doctors possess extensive clinical knowledge regarding the location and extent of lesions, and in such cases, they understand which image or sequence types might be useful for diagnosis. These prior clinical indications can be obtained through manual configuration or machine learning. Furthermore, when the algorithm determines the location and extent of the lesion, it selects specific sequences based on these specific locations and extents. For instance, a pre-defined correspondence table can be set for selecting specific sequences based on the location and extent of the lesion, or a pre-trained model can be used for association; this disclosure is not limited to these methods.

[0049] According to some embodiments, the method 200 can further include, before acquiring the at least one additional image sequence based on the first image sequence: processing the first image sequence based on the first acquisition target to obtain a first lesion type and a first confidence in the first image sequence, wherein the first confidence does not satisfy a confidence threshold. According to such embodiments, acquiring the at least one additional image sequence based on the first image sequence includes acquiring at least one additional image sequence associated with the first lesion type. Thereby, a new additional sequence can be pulled to assist in the identification when the image identification does not satisfy the confidence, so as to obtain enhanced accuracy.

[0050] According to some embodiments, acquiring the at least one additional image sequence associated with the first lesion type can include: determining at least one additional sequence type based on a second sequence type set associated with the first lesion type; and acquiring the at least one additional image sequence pre-stored and associated with the first human body based on the at least one additional sequence type. According to some embodiments, the second sequence type set can include one or more sequence types constructed based on the first lesion type.

[0051] Optionally, the reference sequence set or the reference sequence list can also be generated or updated according to the habits (diagnostic habits) of the doctors or the habits or requirements of the organizations where the doctors are located, etc. According to some embodiments, acquiring the at least one additional image sequence based on the first image sequence can include: acquiring a user identification; acquiring a third sequence type set based on the first acquisition target and the user identification; and acquiring the at least one additional image sequence pre-stored and associated with the first human body based on the third sequence type set. Thereby, different sequence types can be selected according to the user. It can be understood that such embodiments can be combined with the first sequence type set in the above. For example, the sequence type set can be obtained according to both the user identification and the acquisition target and the lesion, etc. Or, such embodiments can also be used independently of each other. It can be understood that the present disclosure is not limited thereto.

[0052] It can be appreciated that throughout this document, the user identification can include at least one of the following: a personal identification and an institutional identification. For example, the user identification can identify the particular physician or other user who needs to analyze the sequence, such as but not limited to an analyst, an operator, a researcher, etc., and the method according to embodiments of the present disclosure can output data specific to the diagnostic habits or diagnostic strategies of that user. Alternatively, the user identification can include the particular medical institution, teaching or research institution, hospital, clinic, association, organization, etc., to which the physician (or other user who needs to analyze the sequence of images) belongs, and the same institution can share similar analysis strategies. As one additional non-limiting example, a particular hospital or medical organization can have a regulation that requires certain sequences to be needed to determine a certain condition, and such data can also be configuration data specific to that hospital or organization.

[0053] For example, the third set of sequence types can be a set of sequence types constructed according to the historical behavior of the user. For example, at least one sequence type associated with the user identification can be obtained by processing the behavior data associated with the user identification. Thus, the sequence types related to the physician can be obtained according to the historical behavior of the physician. The historical behavior can include historical browsing behavior, interaction behavior, operation (or lack of operation), etc., and the present disclosure is not limited thereto. As some specific non-limiting examples, the behavior data associated with the user identification includes the historical interaction degree of the physician for at least one image sequence, and wherein the at least one sequence type associated with the user identification includes the type of sequence whose historical interaction degree is greater than a predetermined interaction degree threshold. For example, the interaction degree can include the viewing time, dwell time, and viewing frequency, etc. of a plurality of medical image sequences (which can include but are not limited to the type of image sequences of "first image sequence" and "take at least one additional image sequence") in the history of the physician. As an example, the interaction degree threshold can be a predetermined viewing duration, such as 1 minute of single viewing or 5 minutes of cumulative viewing, can be a viewing frequency, such as more than three times of selecting the type of sequence for viewing in diagnosis, can be other indicators representing interaction, such as whether there is zooming, rotating, etc. highlighting operation, etc., and it can be appreciated that the present disclosure is not limited thereto.

[0054] According to further non-limiting examples, the behavior data can also include the order correlation of which sequence is looked at first and which sequence is looked at second in history, how the sequences are selected to be grouped, zooming, rotating viewing or other highlighting operations of certain specific sequences, or which sequences are put together for comparison, etc. These correlation features, etc. can be obtained by induction, and optionally, can be obtained by machine learning and neural networks. It can be appreciated that the above are examples, and the present disclosure is not limited thereto.

[0055] As some other non-limiting examples, constructing a set of sequence types based on a user's historical behavior may also include setting a first sequence type in the first image sequence and a third sequence type in the third image sequence as associated sequence types in response to behavioral data associated with a user identifier indicating that a first image sequence and a third image sequence are associated.

[0056] As another optional, non-limiting example, constructing a set of sequence types based on user history behavior could also include obtaining sequence types by analyzing doctor behavior data and extracting priorities for different sequence types from the doctor behavior data. For example, sequence types with more doctor interactions (longer duration, more frequency, or more high-level interactions such as zooming in, comparison operations, etc.) correspond to higher priorities. Once priorities are obtained, they can be used in the control terminal when processing image sequences. For example, in situations with limited bandwidth or data volume, sequences with higher priority can be prioritized for auxiliary recognition and analysis.

[0057] Furthermore, it is understood that although the foregoing descriptions of "sequence type set" or "subset of sequence types" represent some variations of embodiments of this disclosure, the term "set" here is not necessarily understood as a specific form of data storage or combination in the field of data processing, but can include any form of list, queue, cluster, form, table, lookup table, data collection, etc. A sequence type set does not necessarily require that such data be stored in a specific location in the system (e.g., in the form of a table or list), as long as one or more associated sequence types can be obtained based on the current sequence type. As an optional, non-limiting example, if the current sequence type is input to a pre-trained model and one or more image sequences or image sequence types are obtained, the resulting image sequence types can also be referred to as a "sequence type set." It is understood that this disclosure is not limited thereto.

[0058] According to some embodiments, obtaining the at least one additional image sequence based on the first image sequence can comprise: inputting a type of the first image sequence to a pre-trained model to determine at least one additional sequence type; and obtaining a pre-stored at least one additional image sequence associated with the first human body based on the at least one additional sequence type. In such embodiments, the sequence can be selected according to the pre-trained model, e.g., a machine learning model. For example, a number of image sequences can be selected as sample first image sequences, and additional sequences can be selected by a professional (e.g., a doctor) with hand shaking, labeled as sample additional image sequences, and input to a neural network, so that the network can learn the correlation therebetween. When training the model, the input of the model can include patient information, user identification or institution identification to which the doctor belongs, acquisition purpose, acquisition instrument, acquisition stage, etc. The input of the model can also include manually labeled lesion degree, lesion position, pre-analysis result, etc., or these intermediate products of lesion degree, lesion position, analysis result, etc. can also be learned by the model itself. Through such a machine learning model, the correlation between the first image sequence and the additional image sequence can be obtained, e.g., potential correlations that some humans fail to discover, etc.

[0059] According to some embodiments, obtaining the at least one additional image sequence can comprise obtaining the at least one additional image sequence from an image sequence library for the first human body that satisfies at least one of: the image sequences in the image sequence library are acquired based on a second acquisition target different from the first acquisition target; or the image sequences in the image sequence library have different diagnosis stage identifications from the first image sequence.

[0060] In other words, the additional image sequence can be selected from sequences of different acquisition targets, e.g., acquired in different departments, acquired for another lesion, etc. The additional image sequence can also be selected from sequences acquired at different diagnosis stages, e.g., image sequences acquired at initial diagnosis, image sequences acquired at follow-up, etc. As already described in the foregoing, or as will be appreciated by those skilled in the art, the image sequence library here can be stored anywhere, e.g., in the cloud, in a storage card or acquisition card connected to the device, in one or more devices or a combination thereof. For example, image sequences acquired at initial diagnosis can be stored in the cloud, while image sequences acquired at follow-up can be stored in an acquisition card and read together based on a predetermined rule. As another non-limiting example, different departments, different acquisition targets, or different types of sequences can be stored in different terminals, e.g., different acquisition devices or storage devices associated with different acquisition devices or the cloud, etc. It will be appreciated that the present disclosure is not limited thereto.

[0061] According to some embodiments, acquiring the at least one additional image sequence based on the first image sequence can include acquiring a first additional image sequence based on a first resolution, the first resolution being lower than a resolution of the first image sequence. Thereby, image sequences can be pulled with lower resolution, thereby reducing the amount of data to be transmitted and increasing the transmission rate.

[0062] According to some embodiments, the method 200 can further include, in response to determining, based on the first additional image sequence, that there is a first feature for the first acquisition target in the first region, acquiring a second region of a second additional image sequence based on a second resolution, the second resolution being lower than the resolution of the first image sequence, and the second region corresponding to the first region.

[0063] Thereby, after a region is calculated with low resolution, images of the corresponding location can be pulled to assist the analysis. For example, it can not be necessary to pull all images, thereby further reducing the amount of data to be transmitted and increasing the transmission rate. This can be advantageous for some scenarios where the amount of data is large and thus the data transmission is slow, or for small medical institutions where network resources are limited, etc. For example, the low resolution and local pulling function can be manually turned on and off based on user selection, or can be automatically turned on and off according to network bandwidth, data volume, based on the required accuracy, based on the confidence of previous identification, etc.

[0064] As an optional non-limiting example, in the optional example where the method 200 includes processing the first image sequence based on the first acquisition target to obtain a first lesion type and a first confidence in the first image sequence, the first confidence can be compared with a higher confidence threshold and a lower confidence threshold, respectively, if the first confidence only fails to satisfy the higher confidence threshold (i.e., has a relatively high confidence), low resolution can be used to pull other image sequences; if the first confidence fails to satisfy the lower confidence threshold (i.e., has a definitely low confidence), low resolution is not used to pull other image sequences. It can be understood that the above is only an example and the present disclosure is not limited thereto.

[0065] Generally, a diagnosis of a disease is accompanied by screening, re-examination, diagnosis, etc. Multiple image scans can be generated. Doctors can also view multiple images in combination. Software needs to integrate data of multiple different acquisition methods. Then the software calculates the corresponding results. At this time, the doctor compares several different data. Look in combination. According to some embodiments of the present disclosure, effective sequences can be automatically compared or selected according to the lesion to obtain more comprehensive and accurate image processing results. According to one or more embodiments of the present disclosure, corresponding sequences can be intelligently combined / selected according to AI recognition results, sequence analysis results, and doctor's diagnosis habits to assist diagnosis. According to one or more embodiments of the present disclosure, sequences can be reorganized and enhanced recognition sequences can be generated according to the recognition of multiple sequences and according to the disease condition and disease type.

[0066] Furthermore, it can be appreciated that, although the various items of operations are depicted as occurring in a particular sequence in the drawings, this is not intended to be a requirement, and that the operations can occur in other sequences, and not all illustrated operations can be required to achieve the desired results. In addition, it can be appreciated that the various items of operations can be performed by hardware, software, or a combination thereof.

[0067] It can also be appreciated that throughout this disclosure, the image sequence can be or can include two-dimensional image data, or can be or include three-dimensional image data. The image sequence can be image data directly acquired and stored or otherwise transmitted to a terminal device for use by a user. The image sequence can also be processed image data after various image processing. The various image processing can include, but are not limited to, generating post-processing images (curved planar reconstruction, straight planar reconstruction, short axis images, etc. or planar, curved surface projection transformation, etc.), geometric deformation, registration, or can include, but are not limited to, pre-processing (window width and window level adjustment), region selection, overlay, overlay based on target images, etc., or can include other image processing that can be appreciated by those skilled in the art. The image sequence can also be subjected to other analysis processes (e.g., analysis processes for whether there is a lesion feature or a lesion) and contain analysis results (e.g., outlining of a region of interest, segmentation results of tissues, etc.). It can be appreciated that the present disclosure is not limited thereto.

[0068] Figure 4 is a schematic block diagram illustrating an image sequence processing apparatus 400 according to an exemplary embodiment. The image sequence processing apparatus 400 can include a first image sequence acquisition unit 410, an additional image sequence acquisition unit 420, and an image sequence processing unit 430. The first image sequence acquisition unit 410 is configured to acquire a first image sequence, the first image sequence being acquired based on a first acquisition target with respect to a first human body. The additional image sequence acquisition unit 420 is configured to acquire at least one additional image sequence based on the first image sequence, the at least one additional image sequence being with respect to the first human body, and the at least one additional image sequence having a different sequence type from the first image sequence. The image sequence processing unit 430 is configured to process the first image sequence and the at least one additional image sequence based on the first acquisition target to obtain an image processing result related to the first acquisition target.

[0069] It should be understood that Figure 4 The various modules of the apparatus 400 shown in FIG. 4 can correspond to the various steps of the method 200 described with reference to FIG. 2. Thus, the operations, features, and advantages described above for the method 200 apply equally to the apparatus 400 and the modules included therein. For the sake of brevity, certain operations, features, and advantages are not repeated here. Figure 2 The various modules of the apparatus 400 shown in FIG. 4 can correspond to the various steps of the method 200 described with reference to FIG. 2. Thus, the operations, features, and advantages described above for the method 200 apply equally to the apparatus 400 and the modules included therein. For the sake of brevity, certain operations, features, and advantages are not repeated here.

[0070] According to an embodiment of the present disclosure, a computing device is also disclosed, comprising a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the steps of the image sequence processing method according to the embodiments of the present disclosure and the variants thereof.

[0071] According to an embodiment of the present disclosure, a non-transitory computer readable storage medium is also disclosed, having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the image sequence processing method according to the embodiments of the present disclosure and the variants thereof.

[0072] According to an embodiment of the present disclosure, a computer program product is also disclosed, comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the image sequence processing method according to the embodiments of the present disclosure and the variants thereof.

[0073] While specific functions are discussed above with reference to specific modules, it should be noted that the functions of the various modules discussed herein can be split among multiple modules, and / or at least some of the functions of multiple modules can be combined into a single module. A specific module performing an action as discussed herein includes the specific module itself performing the action, or alternatively the specific module invoking or otherwise accessing another component or module that performs the action (or a combination thereof). Thus, a specific module performing an action includes the specific module itself performing the action and / or another module invoked or otherwise accessed by the specific module performing the action. As used herein, the phrase “entity A initiates action B” can mean that entity A issues instructions to perform action B, but entity A itself does not necessarily perform action B. For example, the phrase “the image sequence processing unit 430 is configured to process the first image sequence and the at least one additional image sequence based on the first acquisition target to obtain an image processing result related to the first acquisition target” can mean that the image sequence processing unit 430 instructs other computing terminals, computing cards, cloud computing capabilities, etc. to process the image processing result, and the image sequence processing unit itself does not need to perform the action of processing.

[0074] It should also be understood that various techniques described herein can be described in the general context of software hardware elements or program modules. The above description with respect to Figure 4The various modules described can be implemented in hardware or in hardware combined with software and / or firmware. For example, the modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, the modules can be implemented as hardware logic / circuitry. For example, in some embodiments, one or more of the modules can be implemented together in a System on Chip (SoC). The SoC can include an integrated circuit chip (which includes one or more of a processor (e.g., a Central Processing Unit (CPU), a microcontroller, a microprocessor, a Digital Signal Processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry) and can optionally execute received program code and / or include embedded firmware to perform functions.

[0075] According to an aspect of the present disclosure, there is provided a computing device comprising a memory, a processor, and a computer program stored on the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0076] According to an aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0077] According to an aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0078] In the following, reference will be made to Figure 5 Illustrative examples of the computer device, the non-transitory computer- readable storage medium, and the computer program product are described below.

[0079] Figure 5 An example configuration of a computer device 500 that can be used to implement the methods described herein is shown. For example, Figure 1 The server 120 and / or the client device 110 shown in FIG. 1 can comprise an architecture similar to the computer device 500. The image sequence processing apparatus / device described above can also be implemented, in whole or at least in part, by the computer device 500 or a similar device or system.

[0080] The computer device 500 can be various different types of devices, such as a server of a service provider, a device associated with a client (e.g., a client device), a system on a chip, and / or any other suitable computer device or computing system. Examples of computer device 500 include, without limitation, a desktop computer, a server computer, a laptop or notebook computer, a mobile device (e.g., a tablet computer, a cellular or other wireless phone (e.g., a smart phone), a notepad computer, a mobile station), a wearable device (e.g., glasses, a watch), an entertainment device (e.g., an entertainment appliance, a set-top box, a game console), a television or other display device, an automobile computer, and so forth. Thus, the computer device 500 can range from a full resource device with substantial memory and processor resources (e.g., a personal computer, game console) to a low-resource device with limited memory and / or processing resources (e.g., a traditional set-top box, hand-held game console).

[0081] The computer device 500 can include at least one processor 502, memory 504, communication interface(s) 506, display device 508, other input / output (I / O) devices 510, and one or more mass storage devices 512, which can communicate with one another by way of a system bus 514 or other appropriate communication link.

[0082] The processor 502 can be a single processing unit or a plurality of processing units, all of which can include single or multiple computing cores or processing elements. The processor 502 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor 502 can be configured to fetch and execute computer-readable instructions stored in the memory 504, the mass storage device 512, or any other computer-readable medium, such as program code of an operating system 516, program code of applications 518, program code of other programs 520, and / or the like.

[0083] The memory 504 and mass storage device 512 are examples of computer readable storage media for storing instructions which are executed by the processor 502 to practice the various functionalities described above. By way of example, the memory 504 can generally include both volatile memory and nonvolatile memory (e.g., RAM, ROM, etc.). Further, the mass storage device 512 can generally include hard disk drives, solid state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CD, DVD), storage arrays, network attached storage, storage area networks, etc. Both the memory 504 and the mass storage device 512 can be collectively referred to herein as memory or computer readable storage media, and can be non-transitory media capable of storing computer readable, processor executable program instructions as computer program code which can be executed by the processor 502 as a particular machine configured to implement the operations and functionalities described in the examples herein.

[0084] A number of program modules can be stored on the mass storage device 512. These programs include an operating system 516, one or more application programs 518, other programs 520, and program data 522, and they can be loaded into the memory 504 for execution. Examples of such application programs or program modules can include, for example, computer program logic (e.g., computer program code or instructions) for implementing the client application 112 and / or additional embodiments described herein.

[0085] Although illustrated in Figure 5 the modules 516, 518, 520, and 522, or portions thereof, can be implemented using any form of computer readable media that is accessible by the computer device 500. As used herein, "computer readable media" includes both computer storage media and communication media.

[0086] Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computer device.

[0087] In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. As defined herein, computer storage media does not include communication media.

[0088] The computer device 500 can also include one or more communication interfaces 506 for exchanging data with other devices, such as over a network, direct connection, or the like, as previously discussed. Such communication interfaces can be one or more of: any type of network interface (e.g., network interface card (NIC)), wired or wireless (such as IEEE 802.11 wireless LAN (WLAN)) wireless interface, Worldwide Interoperability Microwave Access (Wi-MAX) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth interface, near field communication (NFC) interface, or the like. The communication interfaces 506 can facilitate communications over a variety of networks and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, and the like. The communication interfaces 506 can also provide communication with external storage devices (not shown), such as storage arrays, network attached storage, storage area networks, and the like. TM

[0089] In some examples, a display device 508, such as a monitor, can be included for displaying information and images to a user. Other I / O devices 510 can be devices that receive various inputs from a user and provide various outputs to the user, and can include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and the like.

[0090] While the present disclosure has been illustrated and described in detail in the drawings and foregoing description, such illustration and description is to be considered illustrative or exemplary and not restrictive; the present disclosure is not limited to the disclosed embodiments. Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed subject matter, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps not listed in the claims, and the word "a" or "an" does not exclude a plurality. The mere fact that measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.​

Claims

1. A method for image sequence processing, comprising: obtaining a first image sequence, the first image sequence being captured for a first human body based on a first capturing target; processing the first image sequence based on the first capturing target to obtain a first lesion type and a first confidence in the first image sequence; in response to determining that the first confidence does not satisfy a confidence threshold, determining at least one additional sequence type from a pre-constructed sequence type set associated with the first lesion type based on the first lesion type; obtaining pre-stored at least one additional image sequence associated with the first human body based on the at least one additional sequence type, the at least one additional image sequence being captured for the first human body and having a different sequence type from the first image sequence; and processing the first image sequence and the at least one additional image sequence based on the first capturing target to obtain an image processing result related to the first capturing target. The first capturing target is for a first lesion, and a first sequence type set associated with the first capturing target includes one or more sequence types constructed based on the first lesion.

2. The method of claim 1, wherein, 3. The method of claim 2, further comprising: before determining the at least one additional sequence type, analyzing the first image sequence based on the first capturing target to determine a lesion position for the first lesion; and wherein a first lesion sequence type set related to the first lesion includes a subset of sequence types respectively related to different lesion positions of the first lesion.

4. The method of claim 2, further comprising, before determining the at least one additional sequence type: analyzing the first image sequence based on the first capturing target to determine a lesion degree for the first lesion; a first lesion sequence type set related to the first lesion includes a subset of sequence types respectively related to different lesion degrees of the first lesion. And wherein, Obtaining the at least one additional image sequence based on the first image sequence includes obtaining at least one additional image sequence associated with the first lesion type.

5. The method of claim 1, wherein, Obtaining the at least one additional image sequence associated with the first lesion type includes:

6. The method of claim 5, wherein, determining at least one additional sequence type based on a second sequence type set associated with the first lesion type; obtaining pre-stored at least one additional image sequence associated with the first human body based on the at least one additional sequence type. The second sequence type set includes one or more sequence types constructed based on the first lesion type.

7. The method of claim 6, wherein, Obtaining the at least one additional image sequence based on the first image sequence includes:

8. The method of claim 1, wherein, obtaining a user identification; obtaining a third sequence type set based on the first capturing target and the user identification; and obtaining pre-stored at least one additional image sequence associated with the first human body based on the third sequence type set. Obtaining the at least one additional image sequence based on the first image sequence includes:

9. The method of claim 1, wherein, inputting a type of the first image sequence to a pre-trained model to determine the at least one additional sequence type; and ​ obtaining at least one additional image sequence associated with the first human based on the at least one additional sequence type.

10. The method of any one of claims 1-9, wherein, obtaining at least one additional image sequence comprises obtaining at least one additional image sequence from an image sequence library for the first human that satisfies at least one of: image sequences in the image sequence library are acquired based on a second acquisition target different from the first acquisition target; or image sequences in the image sequence library have different diagnostic stage identifications from the first image sequence.

11. The method of any one of claims 1-9, wherein, obtaining at least one additional image sequence based on the first image sequence comprises obtaining a first additional image sequence based on a first resolution, the first resolution being lower than a resolution of the first image sequence.

12. The method of claim 11, further comprising: in response to determining, based on the first additional image sequence, that a first feature for the first acquisition target is present in a first region; obtaining a second region of a second additional image sequence based on a second resolution, the second resolution being lower than the resolution of the first image sequence, and the second region corresponding to the first region.

13. An image sequence processing apparatus, comprising: a first image sequence obtaining unit configured to obtain a first image sequence, the first image sequence being acquired based on a first acquisition target for a first human; an additional image sequence obtaining unit configured to process the first image sequence based on the first acquisition target to obtain a first lesion type and a first confidence in the first image sequence; in response to determining that the first confidence does not satisfy a confidence threshold, based on the first lesion type, determine at least one additional sequence type from a pre-constructed sequence type set associated with the first lesion type, and obtain at least one additional image sequence pre-stored and associated with the first human based on the at least one additional sequence type, the at least one additional image sequence being for the first human and having a different sequence type from the first image sequence; and an image sequence processing unit configured to process the first image sequence and the at least one additional image sequence based on the first acquisition target to obtain an image processing result related to the first acquisition target.

14. A computing device, comprising: a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the steps of the method of any one of claims 1-12.

15. A non-transitory computer readable storage medium having stored thereon a computer program, wherein, the computer program, when executed by the processor, implements the steps of the method of any one of claims 1-12.

16. A computer program product comprising a computer program, wherein, the computer program, when executed by the processor, implements the steps of the method of any one of claims 1-12. the computer program, when executed by the processor, implements the steps of the method of any one of claims 1-12.

Images