Image sequence processing method and apparatus, computing device, and storage medium

Abstract

A method, apparatus, computing device, and storage medium for image sequence processing are provided. The method can include: obtaining a first image sequence, the first image sequence being captured based on a first capturing target for a first human body; obtaining at least one additional image sequence based on the first image sequence, the at least one additional image sequence being for the first human body, and the at least one additional image sequence having a different sequence type from the first image sequence; and controlling a terminal to display the first image sequence and the at least one additional image sequence.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and in particular to an image sequence processing method, apparatus, computing device, and storage medium. Background Technology

[0002] Currently, doctors and other analysts often use medical image sequences obtained from medical scanning equipment (such as MRI scanners, magnetic resonance imaging scanners, and computed tomography scanners) for medical analysis or diagnosis. When diagnosing diseases or lesions, doctors often need to select, compare, and combine various different sequence types for analysis, and may need to spend a considerable amount of time sifting through numerous image sequences to find medical image sequences or sequence combinations that clearly reflect the symptoms, resulting in low efficiency in selecting effective medical image sequences.

[0003] Additionally, there may be situations where a doctor is currently reviewing an image sequence of a patient acquired using a certain acquisition device (e.g., CT) or acquisition method, but a more accurate and effective diagnosis might require combining other medical image sequences (e.g., images acquired using MR). In such cases, doctors often need to rely on personal experience to determine different sequence types from the patient's historical images, manually select and retrieve image sequences, and perform further diagnosis, which consumes considerable diagnostic time and may reduce diagnostic efficiency.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] According to one aspect of this disclosure, an image sequence processing method is provided, comprising: acquiring a first image sequence, the first image sequence being acquired for a first human body based on a first acquisition target; acquiring at least one additional image sequence based on the first image sequence, the at least one additional image sequence being for the first human body, and the at least one additional image sequence having a different sequence type from the first image sequence; and controlling a terminal to display the first image sequence and the at least one additional image sequence.

[0006] According to another aspect of this disclosure, an image sequence processing apparatus is provided, comprising: a first image sequence acquisition unit for acquiring a first image sequence, the first image sequence being acquired for a first human body based on a first acquisition target; an additional image sequence acquisition unit for acquiring at least one additional image sequence based on the first image sequence, the at least one additional image sequence being for the first human body, and the at least one additional image sequence having a different sequence type from the first image sequence; and a display unit for controlling a terminal to display the first image sequence and the at least one additional image sequence.

[0007] According to another aspect of this disclosure, a computing device is provided, comprising: a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement an image sequence processing method according to one or more embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements an image sequence processing method according to one or more embodiments of this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, implements an image sequence processing method according to one or more embodiments of this disclosure.

[0010] These and other aspects of this disclosure will be apparent from the embodiments described below, and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0011] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0012] Figure 1 This is a schematic diagram illustrating an example system in which the various methods described herein may be implemented according to exemplary embodiments;

[0013] Figure 2 This is a flowchart illustrating an image sequence processing method according to an exemplary embodiment;

[0014] Figure 3A This is a flowchart illustrating some steps of an image sequence processing method according to another exemplary embodiment;

[0015] Figure 3B This is a schematic diagram illustrating the layout displayed on a terminal according to an exemplary embodiment;

[0016] Figure 3C This is a schematic diagram illustrating the layout displayed on a terminal according to another exemplary embodiment;

[0017] Figure 4 This is a schematic block diagram illustrating an image sequence processing apparatus according to an exemplary embodiment;

[0018] Figure 5 This is a block diagram illustrating an exemplary computer device that can be applied to an exemplary embodiment. Detailed Implementation

[0019] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0020] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. As used herein, the term "multiple" means two or more, and the term "based on" should be interpreted as "at least partially based on". Furthermore, the terms "and / or" and "at least one of..." cover any one of the listed items and all possible combinations thereof.

[0021] Exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0022] Figure 1 This is a schematic diagram illustrating an example system 100 in which various methods described herein may be implemented according to exemplary embodiments.

[0023] refer to Figure 1 The system 100 includes a client device 110, a server 120, and a network 130 that communicatively couples the client device 110 and the server 120.

[0024] Client device 110 includes a display 114 and a client application (APP) 112 that can be displayed on the display 114. Client application 112 can be an application that needs to be downloaded and installed before running, or a lightweight application (liteapp). If client application 112 is an application that needs to be downloaded and installed before running, client application 112 can be pre-installed on client device 110 and activated. If client application 112 is a mini-app, user 102 can directly run client application 112 on client device 110 without installing it, by searching for client application 112 in the host application (e.g., by the name of client application 112) or scanning the graphic code of client application 112 (e.g., barcode, QR code, etc.). In some embodiments, client device 110 can be any type of mobile computing device, including mobile computers, mobile phones, wearable computing devices (e.g., smartwatches, head-mounted devices including smart glasses, etc.), or other types of mobile devices. In some embodiments, the client device 110 may alternatively be a stationary computer device, such as a desktop computer, server computer, or other type of stationary computer device. In some alternative embodiments, the client device 110 may also be or may include a medical image printing device.

[0025] Server 120 is typically a server deployed by an Internet Service Provider (ISP) or Internet Content Provider (ICP). Server 120 can represent a single server, a cluster of multiple servers, a distributed system, or a cloud server providing basic cloud services (such as cloud databases, cloud computing, cloud storage, and cloud communications). It will be understood that, although... Figure 1 The diagram shows that server 120 communicates with only one client device 110, but server 120 can provide background services to multiple client devices simultaneously.

[0026] Examples of network 130 include combinations of local area networks (LANs), wide area networks (WANs), personal area networks (PANs), and / or communication networks such as the Internet. Network 130 can be wired or wireless. In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to process data exchanged through network 130. Furthermore, encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some of the links. In some embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0027] System 100 may further include image acquisition device 140. In some embodiments, Figure 1 The image acquisition device 140 shown may be a medical scanning device, including but not limited to scanning or imaging devices used in positron emission tomography (PET), positron emission tomography with computerized tomography (PET / CT), single photon emission computed tomography with computerized tomography (SPECT / CT), computed tomography (CT), medical ultrasonograph, nuclear magnetic resonance imaging (NMRI), magnetic resonance imaging (MRI), cardiovascular angiograph (CA), digital radiograph (DR), etc. For example, image acquisition device 140 may include digital subtraction angiography scanner, magnetic resonance angiography scanner, computed tomography angiography scanner, positron emission tomography scanner, positron emission tomography (PET) scanner, single photon emission computed tomography (SPT) scanner, computed tomography scanner, medical ultrasound examination equipment, magnetic resonance imaging (MRI) scanner, digital radiography scanner, etc. Image acquisition device 140 may be connected to a server (e.g., Figure 1 The image data is connected to server 120 (or a separate server of the imaging system, not shown in the figure) to perform image data processing, including but not limited to converting scan data (e.g., converting it into a medical image sequence), compressing it, correcting pixels, and reconstructing it in three dimensions.

[0028] Image acquisition device 140 may be connected to client device 110, for example, via network 130, or otherwise directly connected to client device to communicate with client device.

[0029] Optionally, the system may also include an intelligent computing device or a computing card 150. The image acquisition device 140 may include or be connected (e.g., detachably connected) to such a computing card 150. As an example, the computing card 150 can perform image data processing, including but not limited to conversion, compression, pixel correction, reconstruction, etc. As another example, the computing card 150 can implement an image sequence processing method according to embodiments of the present disclosure.

[0030] The system may also include other components not shown, such as a data storage unit. The data storage unit may be a database, data repository, or other means for data storage, and may include a conventional database, a cloud database, a distributed database, etc. For example, direct image data generated by the image acquisition device 140, or medical image sequences or three-dimensional image data obtained through image processing, may be stored in the data storage unit for subsequent retrieval by the server 120 and client device 110. As an example, obtaining at least one additional image sequence as described in the embodiments of this disclosure may include obtaining at least one additional image sequence from the data storage unit, although this disclosure is not limited thereto. Furthermore, the image acquisition device 140 may also directly provide direct image data or medical image sequences or three-dimensional image data obtained through image processing to the server 120 or client device 110, etc.

[0031] Users can use client device 110 to view acquired images or videos, including preliminary image data or images after analysis and processing, view analysis results, interact with the acquired images or analysis results, input acquisition commands, configuration data, and so on. Client device 110 can send configuration data, commands, or other information to image acquisition device 140 to control the acquisition and data processing of image acquisition device.

[0032] For the purposes of this disclosure's embodiments, Figure 1In the example, client application 112 can be an image sequence management application that provides various functions, such as storage management, indexing, sorting, and classification of acquired image sequences. Correspondingly, server 120 can be a server used in conjunction with the image sequence management application. Server 120 can provide image sequence management services to client application 112 running on client device 110 based on user requests or instructions generated according to embodiments of this disclosure. For example, it can manage image sequence storage in the cloud, store and classify image sequences according to specified indexes (including, but not limited to, sequence type, patient identifier, body part, acquisition target, acquisition stage, acquisition machine, presence of lesions, severity, etc.), and retrieve and provide image sequences to client devices according to specified indexes, etc. Alternatively, server 120 can also provide or allocate such service capabilities or storage space to client device 110, whereby client application 112 running on client device 110 provides corresponding image sequence management services based on user requests or instructions generated according to embodiments of this disclosure, etc. It is understood that the above is only one example, and this disclosure is not limited thereto.

[0033] Figure 2 This is a flowchart illustrating an image sequence processing method 200 according to an exemplary embodiment. Method 200 can be implemented on a client device (e.g., ...). Figure 1 The execution is performed at the client device 110 shown, that is, the execution entity of each step of method 200 can be... Figure 1 The client device 110 shown. In some embodiments, method 200 can be performed on a server (e.g., Figure 1 The method 200 is executed at server 120 (as shown in the figure). In some embodiments, the method 200 may be executed in combination by a client device (e.g., client device 110) and a server (e.g., server 120).

[0034] The steps of method 200 are described in detail below.

[0035] In step 210, a first image sequence is acquired, which is acquired based on a first acquisition target and targeting a first human body.

[0036] At step 220, at least one additional image sequence is obtained based on the first image sequence. The at least one additional image sequence is for the first human body and has a different sequence type from the first image sequence.

[0037] At step 230, the control terminal displays the first image sequence and at least one additional image sequence.

[0038] Using the above method, additional sequences of different types for the same person can be automatically retrieved and displayed, thus achieving an intelligent and proactive sequence retrieval effect. This makes it easier and more effective to assist doctors in their analysis, reducing the extra work doctors need to do to retrieve other sequences themselves.

[0039] It is understood that the executing entity here can be client device 110. For example, client device 110 acquires a first image sequence and at least one additional image sequence and displays these image sequences. Alternatively, the executing entity can be different client devices 110, such as one device acquiring the first image sequence (from itself, from another device, or from cloud storage, etc.), another device performing analysis and calculations based on the first image sequence and acquiring additional image sequences, and then controlling another device (“terminal”) to display them. As another example, the executing entity here can be server 120, or it can be computing card 150 included in or connected to image acquisition device 140 or client device 110, etc. Server 120 or computing card 150 can be connected to the desired terminal device in various ways (wired or wireless) to control the terminal device to perform corresponding processing. It is understood that this disclosure is not limited thereto.

[0040] Furthermore, it is understood that "acquiring the first image sequence" does not necessarily mean actively pulling or retrieving the first image sequence, but can include various steps that make the first image sequence exist, including but not limited to receiving pushes from the acquisition device or other devices to obtain the first image sequence, or the first image sequence was originally stored locally, etc.

[0041] Additional sequences can be obtained based on various rules, and some non-limiting example implementations will be given below.

[0042] According to some embodiments, acquiring at least one additional image sequence based on a first image sequence may include: determining at least one additional sequence type based on a first set of sequence types associated with a first acquisition target; and acquiring at least one pre-stored additional image sequence associated with a first human body based on the at least one additional sequence type. In other words, additional image sequences can be acquired based on sequence types associated with the acquisition target. The acquisition target may include a broad or general acquisition target, such as acquisition from a specific department ("from cardiology"), acquisition targeting a specific body part ("lung imaging"), etc. The acquisition target may also include more specific acquisition targets, such as acquisition targeting specific lesions, lesions, or signs, such as, but not limited to, "pneumonic images of the lungs," "whether the patient has pneumonia," etc.

[0043] According to some embodiments, the first acquisition target may be a first lesion, and the first sequence type set may include one or more sequence types constructed based on the first lesion. For example, sequence types that need to be viewed and associated with each lesion in one or more lesions have been pre-constructed. When the first acquisition target is associated with the first lesion in the one or more lesions, one or more other sequence types can be obtained based on the pre-constructed or pre-stored sequence type set, and if corresponding image sequences have been stored or acquired for the patient, the corresponding one or more image sequences can be retrieved (e.g., "actively retrieved") for display. Thus, more accurate and convenient image sequences for doctors to view can be obtained for specific lesion types.

[0044] According to embodiments of this disclosure, suitable sequences can be selected based on the target of collection or the condition. "Selection" can be understood as setting a template for intelligent display based on lesions, allowing different lesions to be viewed using specific sequence combinations. For example, there can be a preset list of optimal reference sequences for each disease, indicating that lesion 1 can view sequences a and b, lesion 2 can view sequences b and c, etc., and when lesion 1 is detected, sequences a and b are selected for display, for example, a simple display or a combined display as described in other embodiments of this disclosure.

[0045] According to some embodiments, method 200 may further include analyzing a first image sequence based on a first acquisition target to determine the lesion location for a first lesion before determining at least one additional sequence type. In such embodiments, the set of first lesion sequence types associated with the first lesion may include subsets of sequence types respectively associated with different lesion locations of the first lesion. Thus, different sequences can be selected based on the lesion location.

[0046] Additional or alternative land, see Figure 3A Method 200 may further include step 300, which further includes sub-steps 310-330. At step 310, lesion detection processing is performed based on the first image sequence to obtain detection results. At step 320, an experience base is queried based on the detection results to obtain associated sequences as one or more additional sequences. For example, the experience base may include a set of diagnostic reference sequence types, such as ultrasound sequences, CT sequences, MR sequences, etc., for a specific lesion. It is understood that the above are merely examples, and this disclosure is not limited thereto.

[0047] At step 330, the acquired additional sequence is combined and integrated with the first image sequence. It is understood that step 300 can be performed after step 220 of method 200 and before step 230, or it can replace step 220.

[0048] According to some embodiments, method 200 may further include analyzing a first image sequence based on a first acquisition target to determine the lesion severity for a first lesion before determining at least one additional sequence type. In such embodiments, the set of first lesion sequence types associated with the first lesion may include subsets of sequence types respectively associated with different lesion severityes of the first lesion. Thus, different sequences can be selected according to the lesion severity. The lesion severity may include the size, color, shape, etc., of the lesion area or sign features, or may include several levels determined based on the lesion or sign in the image, which can be obtained through image analysis algorithms understood by those skilled in the art, and will not be elaborated further here.

[0049] For example, doctors possess extensive clinical knowledge regarding the location and extent of lesions, and in such cases, they need to carefully examine which images to refer to. These prior clinical indications can be obtained through manual configuration or machine learning. Furthermore, when the algorithm determines the location and extent of the lesion, it selects specific sequences based on these specific locations and extents. For instance, a pre-defined correspondence table can be set for selecting specific sequences based on the location and extent of the lesion, or a pre-trained model can be used for association; this disclosure is not limited to these methods.

[0050] Optionally, a reference sequence set or list can be generated or updated based on the doctor's habits (diagnostic habits). According to some embodiments, obtaining at least one additional image sequence based on a first image sequence may include: obtaining a user identifier; obtaining a second sequence type set based on a first acquisition target and the user identifier; and obtaining at least one pre-stored additional image sequence associated with a first human body based on the second sequence type set. Thus, different sequence types can be selected according to the doctor's preference. It is understood that such embodiments can be combined with the first sequence type set mentioned above. For example, a sequence type set related to both the lesion (and optionally further, lesion type or degree, etc.) and the user identifier can be obtained based on both the user identifier and the acquisition target and lesion, etc. Alternatively, such embodiments can be used independently of each other. It is understood that this disclosure is not limited thereto.

[0051] It is understood that throughout this document, user identifiers may include at least one of the following: personal identifiers and institutional identifiers. For example, a user identifier may identify a particular doctor or other user requiring sequence analysis, such as, but not limited to, an analyst, operator, researcher, etc., and the method according to embodiments of this disclosure may output data specific to that user's diagnostic habits or diagnostic strategy. Alternatively, a user identifier may include the specific medical institution, teaching or research institution, hospital, clinic, association, organization, etc., to which the doctor (or other user requiring analysis of reference image sequences) belongs, and the same institution may share similar analysis strategies. As an additional, non-limiting example, a particular hospital or medical organization may specify that certain sequences are required to diagnose a particular condition, and such data may also serve as configuration data specific to that hospital or organization.

[0052] According to some embodiments, the second set of sequence types may be a set of sequence types constructed based on a user's historical behavior. For example, at least one sequence type associated with a user identifier can be obtained by processing behavioral data associated with that user identifier. Thus, sequence types related to a doctor can be obtained based on the doctor's historical behavior. Historical behavior may include historical browsing behavior, interaction behavior, actions (or no actions), etc., and this disclosure is not limited thereto.

[0053] As specific, non-limiting examples, behavioral data associated with a user identifier includes the doctor's historical interaction with at least one image sequence, wherein at least one sequence type associated with the user identifier includes sequences whose historical interaction is greater than a predetermined interaction threshold. For example, interaction may include the doctor's viewing time, dwell time, and viewing frequency for multiple medical image sequences (which may include, but are not limited to, image sequences of the type "first image sequence" and "take at least one additional image sequence") in their history. As examples, the interaction threshold may be a predetermined viewing duration, such as 1 minute for a single viewing or 5 minutes of cumulative viewing; it may be the viewing frequency, such as selecting a sequence of that type more than three times during diagnosis; or it may be other indicators characterizing interaction, such as whether there are emphasized operations like zooming or rotating; and it is understood that this disclosure is not limited thereto.

[0054] According to other non-limiting examples, behavioral data may also include historical sequences viewed in order, how sequences were grouped, zooming in, rotating, or other emphasizing operations on specific sequences, or comparing different sequences together. These association features may be obtained through induction, and optionally, through machine learning and neural networks. It is understood that the above are examples, and this disclosure is not limited thereto.

[0055] As some other non-limiting examples, constructing a set of sequence types based on a user's historical behavior may also include setting a first sequence type in the first image sequence and a second sequence type in the second image sequence as associated sequence types in response to behavioral data associated with a user identifier indicating that a first image sequence and a second image sequence are associated.

[0056] As another optional, non-limiting example, constructing a set of sequence types based on user history behavior could also include obtaining sequence types by analyzing doctor behavior data and extracting priorities for different sequence types from the doctor behavior data. For example, sequence types with more doctor interactions (longer duration, more frequency, or more high-level interactions such as zooming in, comparison operations, etc.) would correspond to higher priorities. Once priorities are obtained, they can be used in the display of image sequences on the control terminal, for example, to sort the image sequences as described below.

[0057] Furthermore, it is understood that although the foregoing descriptions of "sequence type set" or "subset of sequence types" represent some variations of embodiments of this disclosure, the term "set" here is not necessarily understood as a specific form of data storage or combination in the field of data processing, but can include any form of list, queue, cluster, form, table, lookup table, data collection, etc. A sequence type set does not necessarily require that such data be stored in a specific location in the system (e.g., in the form of a table or list), as long as one or more associated sequence types can be obtained based on the current sequence type. As an optional, non-limiting example, if the current sequence type is input to a pre-trained model and one or more image sequences or image sequence types are obtained, the resulting image sequence types can also be referred to as a "sequence type set." It is understood that this disclosure is not limited thereto.

[0058] According to some embodiments, obtaining at least one additional image sequence based on a first image sequence may include: inputting the type of the first image sequence into a pre-trained model to determine at least one additional sequence type; and obtaining at least one pre-stored additional image sequence associated with a first human body based on the at least one additional sequence type. In such embodiments, sequences can be selected based on a pre-trained model, such as a machine learning model. For example, some image sequences can be selected as sample first image sequences, and additional sequences can be selected by hand by a professional (e.g., a doctor), labeled as sample additional image sequences, and input into a neural network so that the network can learn the association between them. When training the model, the model input may include patient information, user identifier or the institution identifier of the doctor, collection purpose, collection instrument, collection stage, etc. The model input may also include manually labeled lesion degree, lesion location, pre-analysis results, etc., or these intermediate products such as lesion degree, lesion location, and analysis results may also be learned by the model itself. Through such a machine learning model, the association between the first image sequence and the additional image sequence can be obtained, such as some potential associations that humans have not discovered.

[0059] According to some embodiments, acquiring at least one additional image sequence may include acquiring at least one additional image sequence from an image sequence library for a first human body that satisfies at least one of the following: the image sequence in the image sequence library is acquired based on a second acquisition target different from the first acquisition target; or the image sequence in the image sequence library has a different diagnostic stage identifier than the first image sequence.

[0060] In other words, the additional image sequences can be selected from sequences acquired for different acquisition targets, such as those acquired in different departments, or those acquired for different lesions. The additional image sequences can also be selected from sequences acquired at different diagnostic stages, such as image sequences acquired during initial and follow-up visits. As described above, or as those skilled in the art will understand, the image sequence library described herein can be stored in any location, such as the cloud, a memory card or acquisition card connected to the device, one or more devices, or a combination thereof. For example, image sequences acquired during an initial visit can be stored in the cloud, while image sequences acquired during a follow-up visit can be stored in an acquisition card and read together based on predetermined rules. As another non-limiting example, sequences from different departments, different acquisition targets, or different types can be stored on different terminals, such as different acquisition devices or storage devices or the cloud associated with different acquisition devices. It is understood that this disclosure is not limited thereto.

[0061] According to some embodiments, obtaining at least one additional image sequence based on a first image sequence may include obtaining the first additional image sequence based on a first resolution. The first resolution is lower than the resolution of the first image sequence. Therefore, it is possible to retrieve the image sequence using a lower resolution, thereby reducing the amount of data transmitted and increasing the transmission rate.

[0062] According to some additional optional embodiments, method 200 may further include: in response to determining, based on a first additional image sequence, the presence of a first feature for a first acquisition target in a first region; and acquiring, based on a second resolution, a second region of a second additional image sequence, the second resolution being lower than the resolution of the first image sequence, and the second region corresponding to the first region.

[0063] Therefore, after calculating a low-resolution region, the image at the corresponding location can be retrieved. For example, it is not necessary to retrieve the entire image, thereby further reducing the amount of data transmitted and increasing the transmission rate. This is highly advantageous for scenarios where large amounts of data result in slow data transmission, or for small medical institutions with limited network resources. For example, the low-resolution and partial retrieval functions can be manually enabled and disabled based on user selection, or automatically enabled and disabled based on network bandwidth, data volume, etc.

[0064] According to some additional or alternative embodiments, method 200 may further include generating a layout of the first image sequence and at least one additional image sequence before the control terminal displays the first image sequence and at least one additional image sequence. In such embodiments, the control terminal displaying the first image sequence and at least one additional image sequence includes a control terminal display layout. Figure 3B An example layout is shown, in which different image sequences can be displayed on a single interface. It is understood that throughout this disclosure, image sequences can be or may include two-dimensional image data, or can be or may include three-dimensional image data. Image sequences can be image data that is directly acquired and stored or otherwise transmitted to a terminal device for user use. Image sequences can also be processed image data after various image processing techniques. These various image processing techniques can include, but are not limited to, generating post-processed images (surface reconstruction, straightening reconstruction, short axis diagrams, or planar / surface projection transformations, etc.), geometric deformation, registration, or can include, but are not limited to, preprocessing (window width and window level adjustment), region selection, overlay, target image-based overlay, etc., or can include other image processing techniques that are understandable to those skilled in the art. Image sequences can also undergo other analytical processes (e.g., analysis of the presence of lesion features or lesions) and include analytical results (e.g., region of interest delineation, tissue segmentation results, etc.). It is understood that this disclosure is not limited thereto.

[0065] Generating a display layout may include loading a predetermined layout template and loading one or more images from a sequence into the layout template. Generating a display layout may include aligning, overlaying, locally enhancing, and arranging the image sequence according to a predetermined strategy, and some specific exemplary embodiments will be described below.

[0066] According to some embodiments, generating the layout of the first image sequence and at least one additional image sequence may include, in response to determining that a first lesion region exists in at least one of the first image sequence and at least one additional image sequence, performing at least one of the following operations:

[0067] a) Enhancement is performed on a region related to the first lesion region in at least one of the first image sequence and at least one additional image sequence. This allows for local enhancement, particularly enhancement of the lesion region or region of interest.

[0068] (b) Align at least two image sequences from the first image sequence and at least one additional image sequence, based at least on the location of the first lesion region. Lesion-based alignment facilitates the correction of errors and offsets in image acquisition and improves user viewing experience.

[0069] c) At least two image sequences from a first image sequence and at least one additional image sequence are overlaid, based at least on the location of the first lesion region. Overlay is performed based on the lesion or region of interest to achieve multifaceted enhancement of the lesion.

[0070] Figure 3C An example of an overlay display is shown. For example... Figure 3C As shown, a superimposed image sequence 353 is obtained by overlaying a vascular image 351 from one image sequence with a brain tissue lesion image 352 from another image sequence based on the lesion region. This process may include three-dimensional mapping processing of the images, such as those processing methods for three-dimensional image sequences known to those skilled in the art, and which will not be elaborated upon here. As shown in the figure, in image 353, the vascular image is restored to reflect the shape of the blood vessels, and the lesion region coincides with the lesion region in the brain tissue lesion image, thereby intuitively displaying the location, characteristics, and extent of the lesion in different image sequences, facilitating viewing by users such as doctors or other personnel.

[0071] It is understood that a)-c) above are merely non-limiting examples of generated layouts, and this disclosure is not limited thereto. For example, the generated layout may also include enhanced display processing of the lesion area, such as highlighting or increasing resolution. As another example, the generated layout may also include selecting only a portion of the lesion area or region of interest, without displaying other parts, thereby achieving a more effective display effect. As another non-limiting example, the generated layout may also include generating an interactive layout, for example, when a user selects an image from any of these integrated sequences, it can return to the original sequence belonging to that image for scrolling and viewing, and so on.

[0072] According to some embodiments, generating the layout of a first image sequence and at least one additional image sequence includes determining the order of the first image sequence and at least one additional image sequence based on a user identifier. For example, basing the order on the user identifier may include selecting an order associated with a doctor, such order being obtained based on the doctor's historical behavior, such as the priority order obtained based on historical behavior as described above. Alternatively, the order may be based on a sequence priority specified by the diagnostic requirements of the user's hospital, diagnostic institution, or other organization. Or, the order may be additionally or alternatively arranged according to a priority order learned by a machine learning model. It will be apparent that this disclosure is not limited thereto.

[0073] Typically, the diagnosis of a disease involves screening, re-examination, and confirmation, resulting in multiple imaging scans. Doctors also review multiple images together. The software needs to integrate data from various acquisition methods and then calculate the corresponding results. At this point, the doctor compares several different data sets for a comprehensive analysis. According to one or more embodiments of this disclosure, the doctor's comparison process can be simplified, facilitating rapid lesion localization. Furthermore, according to some embodiments of this disclosure, effective sequences can be automatically compared or selected based on lesions, and the information can be presented to the doctor in an optimized manner. According to one or more embodiments of this disclosure, images with corresponding sequences can be intelligently combined / selected based on AI recognition results, sequence analysis results, and the doctor's diagnostic habits to assist in diagnosis. According to one or more embodiments of this disclosure, sequences can be recombined and enhanced recognition sequences can be generated based on the identification of multiple sequences and according to the condition and disease type.

[0074] Furthermore, it is understood that although the various operations are depicted in the accompanying drawings in a specific order, this should not be construed as requiring that these operations be performed in the specific order shown or in chronological order, nor should it be construed as requiring that all the operations shown be performed to obtain the desired result.

[0075] Figure 4This is a schematic block diagram illustrating an image sequence processing apparatus 400 according to an exemplary embodiment. The image sequence processing apparatus 400 may include a first image sequence acquisition unit 410, an additional image sequence acquisition unit 420, and a display unit 430. The first image sequence acquisition unit 410 is used to acquire a first image sequence, which is acquired based on a first acquisition target and targeting a first human body. The additional image sequence acquisition unit 420 is used to acquire at least one additional image sequence based on the first image sequence, which is targeted at the first human body, and the at least one additional image sequence has a different sequence type than the first image sequence. The display unit 430 is used to control a terminal to display the first image sequence and the at least one additional image sequence.

[0076] It should be understood that Figure 4 The various modules of the device 400 shown can be connected to the reference. Figure 2 The steps in method 200 described correspond to each other. Therefore, the operations, features, and advantages described above for method 200 also apply to apparatus 400 and its included modules. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0077] According to embodiments of the present disclosure, a computing device is also disclosed, including a memory, a processor, and a computer program stored on the memory, wherein the processor is configured to execute the computer program to implement the steps of the image sequence processing method and variations thereof according to embodiments of the present disclosure.

[0078] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium is also disclosed, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the image sequence processing method and variations thereof according to embodiments of the present disclosure.

[0079] According to embodiments of the present disclosure, a computer program product is also disclosed, including a computer program, wherein when the computer program is executed by a processor, it implements the steps of the image sequence processing method and variations thereof according to embodiments of the present disclosure.

[0080] While specific functions have been discussed above with reference to specific modules, it should be noted that the functions of the modules discussed herein can be divided into multiple modules, and / or at least some functions of multiple modules can be combined into a single module. The specific module performing an action discussed herein includes the specific module itself performing the action, or alternatively, the specific module calling or otherwise accessing another component or module that performs the action (or performs the action in conjunction with the specific module). Therefore, a specific module performing an action can include the specific module performing the action itself and / or another module that the specific module calls or otherwise accesses to perform the action. As used herein, the phrase "Entity A initiates action B" can mean that entity A issues an instruction to perform action B, but entity A itself does not necessarily perform action B. For example, the phrase "Display unit 430 is used to control the terminal to display a first image sequence and at least one additional image sequence" can mean that display unit 430 instructs the terminal (e.g., the display unit may not be located locally on the terminal) to present an additional image sequence, without display unit 430 itself needing to perform the "presentation" action.

[0081] It should also be understood that this article can describe various technologies in the general context of software and hardware components or program modules. The above regarding... Figure 4 The various modules described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these modules can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules can be implemented as hardware logic / circuit. For example, in some embodiments, one or more of the modules can be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip (which includes a processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components of other circuitry) and may optionally execute received program code and / or include embedded firmware to perform functions.

[0082] According to one aspect of this disclosure, a computing device is provided, including a memory, a processor, and a computer program stored in the memory. The processor is configured to execute the computer program to implement the steps of any of the method embodiments described above.

[0083] According to one aspect of this disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the method embodiments described above.

[0084] According to one aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the method embodiments described above.

[0085] In the following text, combined with Figure 5 Illustrative examples describing such computer devices, non-transitory computer-readable storage media, and computer program products.

[0086] Figure 5 An example configuration of a computer device 500 that can be used to implement the methods described herein is shown. For example, Figure 1 The server 120 and / or client device 110 shown may include an architecture similar to computer device 500. The aforementioned image sequence processing device / apparatus may also be implemented wholly or at least partially by computer device 500 or similar devices or systems.

[0087] Computer device 500 can be a variety of different types of devices, such as a service provider's server, a device associated with a client (e.g., a client device), a system-on-a-chip, and / or any other suitable computer device or computing system. Examples of computer device 500 include, but are not limited to: desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablets, cellular or other wireless phones (e.g., smartphones), notebook computers, mobile stations), wearable devices (e.g., glasses, watches), entertainment devices (e.g., entertainment appliances, set-top boxes communicatively coupled to a display device, game consoles), televisions or other display devices, automotive computers, and so on. Therefore, the range of computer device 500 can be from full-resource devices with large amounts of memory and processor resources (e.g., personal computers, game consoles) to low-resource devices with limited memory and / or processing resources (e.g., traditional set-top boxes, handheld game consoles).

[0088] Computer device 500 may include at least one processor 502, memory 504, multiple communication interfaces 506, display device 508, other input / output (I / O) devices 510, and one or more mass storage devices 512 capable of communicating with each other, such as via system bus 514 or other suitable connections.

[0089] Processor 502 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 502 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, processor 502 may be configured to acquire and execute computer-readable instructions stored in memory 504, mass storage device 512, or other computer-readable media, such as program code of operating system 516, program code of application program 518, program code of other program 520, etc.

[0090] Memory 504 and mass storage device 512 are examples of computer-readable storage media for storing instructions executed by processor 502 to perform the various functions described above. For example, memory 504 may generally include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, mass storage device 512 may generally include hard disk drives, solid-state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Both memory 504 and mass storage device 512 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 502 as a specific machine configured to perform the operations and functions described in the examples herein.

[0091] Multiple program modules may be stored on mass storage device 512. These programs include operating system 516, one or more applications 518, other programs 520, and program data 522, and they may be loaded into memory 504 for execution. Examples of such applications or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing components / functions such as client application 112 and / or other embodiments described herein.

[0092] Although Figure 5 The modules 516, 518, 520, and 522, or portions thereof, are illustrated as being stored in memory 504 of computer device 500; however, modules 516, 518, 520, and 522 may be implemented using any form of computer-readable medium accessible by computer device 500. As used herein, “computer-readable medium” includes at least two types of computer-readable media: computer storage media and communication media.

[0093] Computer storage media includes volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, DVD, or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other non-transfer medium that can be used to store information for access by computer equipment.

[0094] In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data within modulated data signals such as carrier waves or other transmission mechanisms. Computer storage media as defined herein do not include communication media.

[0095] Computer device 500 may also include one or more communication interfaces 506 for exchanging data with other devices, such as via a network, direct connection, etc., as discussed above. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), wired or wireless (such as IEEE 802.11 Wireless LAN (WLAN)) wireless interface, Wi-MAX interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth. TM Interfaces include near-field communication (NFC) interfaces. Communication interface 506 facilitates communication across various network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. Communication interface 506 can also provide communication with external storage devices (not shown) such as storage arrays, network-attached storage, storage area networks, etc.

[0096] In some examples, a display device 508, such as a monitor, may be included for displaying information and images to the user. Other I / O devices 510 may be devices that receive various inputs from the user and provide various outputs to the user, and may include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and so on.

[0097] Although this disclosure has been described and illustrated in detail in the accompanying drawings and the foregoing description, such description and illustration should be considered illustrative and suggestive, not restrictive; this disclosure is not limited to the disclosed embodiments. By studying the drawings, the disclosure, and the appended claims, those skilled in the art will be able to understand and implement variations of the disclosed embodiments in practicing the claimed subject matter. In the claims, the word "comprising" does not exclude other elements or steps not listed, and the words "a" or "an" do not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be beneficial.

Claims

1. An image sequence processing method, comprising: Acquire a first image sequence, the first image sequence being acquired based on a first acquisition target targeting a first human body, the first acquisition target targeting a first lesion; Based on a first set of sequence types associated with the first acquisition target, at least one additional sequence type is determined that is different from the first image sequence in the first set of sequence types, wherein the first set of sequence types includes one or more sequence types constructed based on the first lesion, and wherein the first lesion sequence type set associated with the first lesion includes a subset of sequence types associated with different lesion locations of the first lesion; Based on the at least one additional sequence type, at least one pre-stored additional image sequence associated with the first human body is obtained, wherein the at least one additional image sequence is specific to the first human body and has a different sequence type from the first image sequence; and The control terminal displays the first image sequence and the at least one additional image sequence.

2. The method of claim 1, further comprising, before determining at least one additional sequence type: analyzing the first image sequence based on the first acquisition target to determine the degree of lesion for the first lesion; And among them, The set of first lesion sequence types associated with the first lesion includes subsets of sequence types that are respectively associated with different lesion degrees of the first lesion.

3. The method according to claim 1, wherein, Obtaining at least one additional image sequence based on the first image sequence includes: Obtain user identifier; A second sequence type set is obtained based on the first acquisition target and the user identifier; and Based on the second set of sequence types, at least one pre-stored additional image sequence associated with the first human body is obtained.

4. The method according to claim 3, wherein, The second set of sequence types is a set of sequence types constructed based on the user's historical behavior.

5. The method according to claim 1, wherein, Obtaining at least one additional image sequence based on the first image sequence includes: The type of the first image sequence is input into a pre-trained model to determine at least one additional sequence type; and Based on the at least one additional sequence type, obtain at least one pre-stored additional image sequence associated with the first human body.

6. The method according to any one of claims 1-5, wherein, Obtaining at least one additional image sequence includes obtaining at least one additional image sequence from a library of image sequences for the first human body that satisfy at least one of the following: The image sequences in the image sequence library were acquired based on a second acquisition target different from the first acquisition target; or The image sequences in the image sequence library have different diagnostic stage identifiers than the first image sequence.

7. The method according to any one of claims 1-5, wherein, Obtaining at least one additional image sequence based on the first image sequence includes obtaining a first additional image sequence based on a first resolution, wherein the first resolution is lower than the resolution of the first image sequence.

8. The method according to claim 7, further comprising: In response to determining, based on the first additional image sequence, the presence of a first feature for the first acquisition target in the first region; A second region of a second additional image sequence is obtained based on a second resolution, the second resolution being lower than the resolution of the first image sequence, and the second region corresponding to the first region.

9. The method according to any one of claims 1-5, further comprising generating a layout of the first image sequence and the at least one additional image sequence before the control terminal displays the first image sequence and the at least one additional image sequence, wherein, The control terminal displays the first image sequence and the at least one additional image sequence, including the layout displayed by the control terminal.

10. The method according to claim 9, wherein, Generating the layout of the first image sequence and the at least one additional image sequence includes, in response to determining that a first lesion region exists in at least one of the first image sequence and the at least one additional image sequence, performing at least one of the following operations: Enhancement is performed on at least one image sequence in the first image sequence and the at least one additional image sequence that is related to the first lesion region. At least two of the first image sequences and the at least one additional image sequence are aligned based on the location of the first lesion region. or At least two of the first image sequences and the at least one additional image sequence are superimposed, based at least on the location of the first lesion region.

11. The method according to claim 9, wherein, Generating the layout of the first image sequence and the at least one additional image sequence includes determining the order of the first image sequence and the at least one additional image sequence based on a user identifier.

12. An image sequence processing apparatus, comprising: The first image sequence acquisition unit is used to acquire a first image sequence, wherein the first image sequence is acquired based on a first acquisition target targeting a first human body, and the first acquisition target targets a first lesion. An additional image sequence acquisition unit is configured to, based on a first sequence type set associated with the first acquisition target, determine at least one additional sequence type in the first sequence type set that is different from the first image sequence, wherein the first sequence type set includes one or more sequence types constructed based on the first lesion, and wherein the first lesion sequence type set associated with the first lesion includes subsets of sequence types respectively associated with different lesion locations of the first lesion; and acquire, based on the at least one additional sequence type, at least one pre-stored additional image sequence associated with the first human body, wherein the at least one additional image sequence is for the first human body, and the at least one additional image sequence has a different sequence type from the first image sequence; and The display unit is used to control the terminal to display the first image sequence and the at least one additional image sequence.

13. A computing device, comprising: Memory, processor, and computer program stored on said memory, The processor is configured to execute the computer program to implement the steps of the method according to any one of claims 1-11.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-11.

15. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-11.

Images