Image detection method, device, electronic equipment and system
By establishing a heterogeneous data cache mechanism between the user terminal device and the AI device, using the PACS device to receive and forward the request, and mapping the two-dimensional image into three-dimensional image for calculation, the AI system cannot process the concurrency of multiple terminal devices at the same time. The bandwidth problem of requesting and repeated transmission of images realizes real-time calculation and display of images, reducing network transmission pressure.
Patent Information
- Application Number
- CN202111234232.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-10-22
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, AI systems cannot process concurrent requests from multiple terminal devices at the same time, resulting in delays in analysis results, and when medical staff measure the same lesion image multiple times, it is necessary to repeatedly transmit the marked images, increasing the network. Transmission pressure.
By establishing a heterogeneous data cache mechanism between the user terminal device and the AI device, using the PACS device to receive the request of the terminal device and forward it to the AI device, and map the two-dimensional image into three-dimensional image for calculation in advance, reducing the response time of the AI device , and reduces bandwidth loss of repeated transmission images.
Realize real-time calculation and display of images under the premise of reducing network bandwidth usage, shorten the time from issuing user measurement instructions to obtain results, avoiding the problem that AI devices cannot handle concurrent requests from multiple terminal devices at the same time, and alleviating the problem of network transmission pressure.
Smart Images

Figure CN113870981B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image detection method, device, electronic equipment and system. Background Art
[0002] In clinical work, an artificial intelligence (AI) system is needed to process lesion images, which can quickly and accurately assist medical staff in obtaining characteristic information of the lesion site and reduce errors caused by manual line drawing and measurement.
[0003] However, because the existing technology consumes a lot of computing resources when processing lesion images, the terminal device can only download and view the lesion image from the image storage device, and send the lesion image and measurement information directly to the AI system. After the calculation is completed in an offline manner, the analysis result is returned. However, the AI system cannot process and respond to concurrent requests from multiple terminal devices at the same time, and the complete offline calculation also takes a certain amount of time, which delays the time to obtain the analysis results. In addition, when medical staff need to measure the same lesion image multiple times, the existing technology also needs to repeatedly transmit the marked lesion image multiple times, which puts a lot of pressure on the network transmission. Summary of the invention
[0004] The purpose of the present application includes, for example, providing an image detection method, device, electronic device and system, which can realize real-time calculation and display of images while reducing the occupied network bandwidth.
[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:
[0006] In a first aspect, an embodiment of the present application provides an image detection method, the method comprising:
[0007] Receiving an image sequence sent by an image archiving and communication device, the image sequence including a plurality of two-dimensional image frames, the plurality of two-dimensional image frames including: an image displayed on a terminal device;
[0008] Mapping the multiple frames of two-dimensional images into three-dimensional images respectively;
[0009] Receiving image selection information sent by the image archiving and communication device, the image selection information including: an identifier of a target two-dimensional image selected by a user on the terminal device and a target two-dimensional coordinate clicked by the user on the target two-dimensional image;
[0010] Determine the real size information of the part where the two-dimensional coordinates of the target are located according to the image selection information and the three-dimensional image corresponding to the two-dimensional image of the target;
[0011] The real size information of the part where the target two-dimensional coordinates are located is sent to the image archiving and communication device for storage, and is sent to the terminal device via the image archiving and communication device.
[0012] In an optional implementation, determining the real size information of the part where the two-dimensional coordinates of the target are located according to the image selection information and the three-dimensional image corresponding to the two-dimensional image of the target includes:
[0013] Determining a three-dimensional image corresponding to the target two-dimensional image according to the identifier of the target two-dimensional image;
[0014] Determining the target three-dimensional coordinates of the target two-dimensional coordinates in the three-dimensional image according to the target two-dimensional coordinates and the three-dimensional image;
[0015] According to the target three-dimensional coordinates, the real size information of the part where the target two-dimensional coordinates are located is determined.
[0016] In an optional implementation, determining the target three-dimensional coordinates of the target two-dimensional coordinates in the three-dimensional image according to the target two-dimensional coordinates and the three-dimensional image includes:
[0017] Determine the distance between the target two-dimensional coordinates and the origin of the image coordinate system where the three-dimensional image is located, and obtain the spatial coordinate system distance;
[0018] The ratio of the spatial coordinate system distance to the image pixel resolution is determined to obtain the target three-dimensional coordinates, wherein the image pixel resolution is the pixel resolution of the target three-dimensional image.
[0019] In an optional implementation manner, determining the real size information of the part where the two-dimensional coordinates of the target are located according to the three-dimensional coordinates of the target includes:
[0020] Determining a three-dimensional area corresponding to the clicked position in the three-dimensional image according to the target three-dimensional coordinates;
[0021] According to the three-dimensional area corresponding to the clicked part in the three-dimensional image, the real size information of the part where the target two-dimensional coordinates are located is determined.
[0022] In an optional implementation, determining the real size information of the part where the target two-dimensional coordinates are located according to the three-dimensional area corresponding to the clicked part in the three-dimensional image includes:
[0023] Calculating the transposition of the three-dimensional region and the transposition matrix to obtain a transposed target part boundary matrix;
[0024] Calculating the product of the transposed target part boundary matrix and the diagonal matrix to obtain the transposed diagonal target part boundary matrix, wherein the diagonal matrix is obtained based on the image pixel resolution and the target two-dimensional coordinates;
[0025] Multiplying the transposed diagonal target part boundary matrix with a preset general matrix to obtain a first target part boundary matrix;
[0026] Adding the first target part boundary matrix to a preset general formula to obtain a second target part boundary matrix;
[0027] The boundary matrix of the second target part is inverted to obtain the real size information of the part where the two-dimensional coordinates of the target are located.
[0028] In an optional implementation, mapping the multiple frames of two-dimensional images into three-dimensional images includes:
[0029] Parsing the header files of the multiple frames of two-dimensional images respectively to obtain the position of each of the two-dimensional images in the image sequence;
[0030] According to the position of the two-dimensional image in the image sequence, each of the two-dimensional images is mapped into a three-dimensional image in sequence.
[0031] In an optional embodiment, the method further includes:
[0032] The target two-dimensional coordinates are sent to an image archiving and communication device, and the image archiving and communication device stores the target two-dimensional coordinates.
[0033] In a second aspect, an embodiment of the present application provides an image detection device, the device comprising:
[0034] A receiving module, used for receiving an image sequence sent by an image archiving and communication device, wherein the image sequence includes a plurality of two-dimensional images, and the plurality of two-dimensional images include: an image displayed on a terminal device;
[0035] The receiving module is further used to receive image selection information sent by the image archiving and communication device, the image selection information including: an identifier of a target two-dimensional image selected by a user on the terminal device and a target two-dimensional coordinate clicked by the user on the target two-dimensional image;
[0036] A processing module, used for mapping the multiple frames of two-dimensional images into three-dimensional images respectively;
[0037] The processing module is also used to determine the real size information of the part where the two-dimensional coordinates of the target are located according to the image selection information and the three-dimensional image corresponding to the two-dimensional image of the target;
[0038] The sending module sends the real size information of the part where the target two-dimensional coordinates are located to the terminal device via the image archiving and communication device.
[0039] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the image detection method as described in any one of the aforementioned embodiments.
[0040] In a fourth aspect, an embodiment of the present application provides an image detection system, the system comprising: the electronic device, the image archiving and communication device, and the terminal device as described in the above embodiment;
[0041] The image archiving and communication device is communicatively connected with the electronic device and the terminal device respectively.
[0042] The beneficial effects of the embodiments of the present application include, for example:
[0043] The image detection method, device, electronic device and system provided by the embodiment of the present application are adopted. First, the embodiment of the present application uses PACS (Picture Archiving and Communication Systems) equipment to establish a heterogeneous data cache mechanism between the user terminal device and the AI device, which is used to store image information, store the operation instructions of the terminal device and forward them to the AI device, and store the real size information of the part that the terminal device needs to know and forward it to the terminal device. In other words, the AI device directly receives requests sent by multiple terminal devices at the same time in the prior art, and the embodiment of the present application changes it to a cache and forwarding mechanism in which the AI device sequentially receives requests from multiple terminal devices via the PACS device, thereby avoiding the problem that the AI device in the prior art cannot simultaneously process and respond to concurrent requests from multiple terminal devices. In addition, before receiving the target two-dimensional coordinates selected by the user on the terminal device, the AI device pre-receives multiple frames of two-dimensional images transmitted from the PACS device and maps them into corresponding three-dimensional images, so that in subsequent calculations, the AI device can directly perform calculations on the three-dimensional image, shortening the time from the user's measurement instruction to obtaining the result.
[0044] Secondly, when medical staff use a lesion image for multiple measurements, in the embodiment of the present application, the PACS device receives and forwards the target two-dimensional coordinates clicked by the user on the image displayed on the terminal device, rather than the marked lesion image in the prior art, thereby reducing the bandwidth loss of repeated image transmission. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0046] Figure 1 It is a structural schematic diagram of an image detection system in the prior art;
[0047] Figure 2 A schematic diagram of the structure of the image detection system provided in the embodiment of the present application;
[0048] Figure 3 A schematic diagram of the steps of the image detection method provided in the embodiment of the present application;
[0049] Figure 4 A schematic diagram of another embodiment of the image detection method provided in the embodiment of the present application;
[0050] Figure 5 A schematic diagram of another embodiment of the image detection method provided in the embodiment of the present application;
[0051] Figure 6 A schematic diagram of another embodiment of the image detection method provided in the embodiment of the present application;
[0052] Figure 7 A schematic diagram of another embodiment of the image detection method provided in the embodiment of the present application;
[0053] Figure 8 A schematic diagram of another embodiment of the image detection method provided in the embodiment of the present application;
[0054] Fig. 9 A schematic diagram of the structure of an image detection device provided in an embodiment of the present application;
[0055] Fig.10 A schematic diagram of the structure of an electronic device corresponding to the image detection method provided in an embodiment of the present application.
[0056] Icons: 101-AI device; 102-terminal device; 103-image archiving and communication device; 104-electronic device; 10-image detection device; 1001-receiving module; 1002-processing module; 1003-sending module; 501-processor; 502-memory. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.
[0058] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0059] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0060] In the description of the present application, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, it is based on the orientation or position relationship shown in the drawings, or is the orientation or position relationship in which the product of the invention is usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.
[0061] It should be noted that, in the absence of conflict, the features in the embodiments of the present application may be combined with each other.
[0062] At present, in clinical work, artificial intelligence systems are needed to process lesion images, obtain characteristic information of lesion sites, and reduce errors caused by manual line measurement by medical staff. Figure 1 As shown, in the prior art, the terminal device 102 can only download and view the lesion image from the image archiving and communication device 103, and send the lesion image and measurement information directly to the AI device 101. After the AI device 101 completes the calculation in an offline manner, it returns the analysis result and displays it on the terminal device 102. This interactive method in the prior art is only applicable to local operations limited to a single computer. When there are a large number of concurrent operations, the AI device cannot simultaneously process and respond to concurrent requests from multiple terminal devices 102, and the complete offline calculation also takes a certain amount of time, which delays the time to obtain the analysis result. In addition, when medical staff need to measure the same lesion image multiple times, the prior art also needs to repeatedly transmit the marked lesion image multiple times, which puts a lot of pressure on network transmission.
[0063] Based on this, the applicant has proposed an image detection method, device, electronic device and system after research. A heterogeneous data cache mechanism is established between the user terminal device and the AI device using the PACS device. The PACS device receives the request of the terminal device and then sends it to the AI device for processing, avoiding the problem that the AI device cannot process and respond to concurrent requests from multiple terminal devices at the same time. In addition, the PACS device can send one or more images displayed on the terminal device to the AI device in advance and map them into corresponding three-dimensional images. In this way, in subsequent calculations, the calculation is directly performed on the three-dimensional image based on the target two-dimensional coordinates clicked by the user on the terminal device, shortening the time from the user's measurement instruction to obtaining the result.
[0064] Figure 2 The schematic diagram of the structure of the image detection system provided in the embodiment of the present application is as follows: Figure 2 As shown, the system may include: an electronic device 104 corresponding to the image detection method, an image archiving and communication device 103, and a terminal device 102, wherein, different from the prior art, the image archiving and communication device 103 is communicatively connected to the electronic device 104 and the terminal device 102, respectively, and serves as a cache device between the electronic device 104 and the terminal device 102.
[0065] The electronic device 104 is used to calculate the measurement result of the icon image according to the measurement mark information generated by the user's operation on the two-dimensional image transmitted by the terminal device 102 and the two-dimensional image transmitted by the image archiving and communication device 103.
[0066] The image archiving and communication device 103 can be used to store the above-mentioned two-dimensional images, and can also be used to store the operation instructions of the terminal device 102 and forward them to the electronic device 104, and store the real size information of the required part calculated by the electronic device 104 and forward it to the terminal device 102.
[0067] The terminal device 102 may be a device that directly interacts with the user. The user may view the two-dimensional images stored in the above-mentioned image archiving and communication device 103 on the terminal device 102 , and may also operate the target two-dimensional image on the terminal device 102 .
[0068] The following is an explanation of an image detection method, device, electronic device and system provided in an embodiment of the present application in combination with a number of specific application examples.
[0069] Figure 3 The following is a schematic diagram of the steps of the image detection method provided in the embodiment of the present application. The execution subject of the method is a remote electronic device with computing and processing capabilities, which may be a cloud server. Figure 3 As shown, the method includes:
[0070] Step S201, receiving an image sequence sent by an image archiving and communication device, the image sequence includes multiple frames of two-dimensional images, and the multiple frames of two-dimensional images include: an image displayed on a terminal device.
[0071] Image archiving and communication equipment is the equipment that runs the PACS system and is used to perform various functions of the PACS system.
[0072] The two-dimensional image may be a non-invasive picture or image of a part of human tissue for medical treatment or medical research, and may include: computed tomography (CT) medical images, magnetic resonance (MR) medical images, which are not limited in this application. In addition, the two-dimensional image may be obtained from a single examination or multiple examinations.
[0073] The image archiving and communication device can be used to store the above-mentioned two-dimensional images, and can also be used to store the operation instructions of the terminal device and forward them to the AI device, as well as store the real size information of the part that the terminal device needs to know and forward it to the terminal device.
[0074] The terminal device may be a device that directly interacts with the user, and the user may view the two-dimensional images stored in the above-mentioned image archiving and communication devices on the device, and may also operate the above-mentioned two-dimensional images on the terminal device.
[0075] The user opens and displays one or more two-dimensional images on the terminal device, and clicks the corresponding button of the AI measurement on the terminal device, for example, smart measurement. After that, the start command triggered by the user clicking the button will be sent by the terminal device to the image archiving and communication device, and the image archiving and communication device will obtain an image sequence composed of one or more two-dimensional images displayed on the terminal device according to the ID (Identity document) and image address of the two-dimensional image displayed on the terminal device, and send it to the artificially preset path of the AI device.
[0076] Step S202: Map multiple frames of two-dimensional images into three-dimensional images respectively.
[0077] Preprocessing is performed on multiple frames of two-dimensional images. The preprocessing method may be data normalization processing, such as grayscale stretching, normalization, etc.; it may also be image filtering processing or image enhancement processing, which is not limited in this application.
[0078] According to the pre-processed data of multiple frames of two-dimensional images, a corresponding three-dimensional image is constructed using a three-dimensional reconstruction algorithm.
[0079] The three-dimensional reconstruction algorithm may be a MC (Marching Cube) algorithm, or a ray casting method, a snowball method, or a shear deformation method, which is not limited in the present application.
[0080] Step S203, receiving image selection information sent by the image archiving and communication device, the image selection information including: the identifier of the target two-dimensional image selected by the user on the terminal device and the target two-dimensional coordinates clicked by the user on the target two-dimensional image.
[0081] After the user clicks the corresponding button for AI measurement on the terminal device, the measurement operation will be performed on the terminal device. For example, the user can click on the center point of a tumor site or organ on the two-dimensional image displayed on the terminal device, or mark the bounding box of the tumor site or organ on the two-dimensional image displayed on the terminal device. It is understandable that when the user clicks on the terminal device, a target two-dimensional coordinate will be generated, and when the user marks the bounding box on the terminal device, multiple target two-dimensional coordinates will be generated. Accordingly, the above-mentioned target two-dimensional coordinates may include a coordinate that can be used to identify a point. Alternatively, it may also include multiple coordinates to identify an area in the target two-dimensional image.
[0082] Optionally, the image selection information may be the coordinates of the user's click operation or marking operation recorded by the terminal device, and the identifier of the target two-dimensional image on which the user performs the operation.
[0083] After the user completes the above measurement operation, the receiving image archiving and communication device will receive the above one or more target two-dimensional coordinates from the terminal device, and send them to the AI device for processing together with the identifier of the target two-dimensional image corresponding to the operation.
[0084] The user can perform the above measurement operation multiple times on the terminal device. Each time the user performs a measurement operation, a set of target two-dimensional coordinates will be sent to the receiving image archiving and communication device, and multiple transmissions will be performed sequentially.
[0085] Step S204, determining the real size information of the part where the target two-dimensional coordinates are located according to the image selection information and the three-dimensional image corresponding to the target two-dimensional image.
[0086] After receiving the image selection information sent by the image archiving and communication device, the AI device uses the image segmentation algorithm to calculate the size information of the corresponding part in the three-dimensional image where the image selection information is located, because the image selection information is the user's annotation of the displayed two-dimensional image on the terminal, and calculates the real size information of the part where the target two-dimensional coordinates are located based on the image selection information. The specific calculation process will be described in detail in the following embodiments. Step S205, the real size information of the part where the target two-dimensional coordinates are located is sent to the image archiving and communication device for storage, and sent to the terminal device via the image archiving and communication device.
[0087] After calculating the actual size information of the target two-dimensional coordinates, the AI device sends the information to the image archiving and communication device. The image archiving and communication device stores the received actual size information of the target two-dimensional coordinates and then forwards it to the terminal device for display.
[0088] In this embodiment, the image archiving and communication device establishes a heterogeneous data caching mechanism between the user terminal device and the AI device, avoiding the problem that the AI device cannot simultaneously process and respond to concurrent requests from multiple terminal devices. The caching mechanism established by the image archiving and communication device can also send one or more images displayed on the terminal device to the AI device in advance and map them into corresponding three-dimensional images. In this way, in subsequent calculations, calculations are performed directly on the three-dimensional image, shortening the time from the user's measurement instruction to the result. In addition, the image archiving and communication device forwards the coordinates of the user's operation on the two-dimensional image at the front end to the AI device, reducing the bandwidth loss of repeated image transmission.
[0089] like Figure 4 As shown, optionally, in the above step S202, mapping multiple frames of two-dimensional images into three-dimensional images respectively can be obtained by the following steps S2021 to S2022.
[0090] Step S2021, respectively parse the header files of multiple frames of two-dimensional images to obtain the position of each two-dimensional image in the image sequence.
[0091] Optionally, the header file in the two-dimensional image may include, but is not limited to: the ID of the two-dimensional image, the image pixel resolution of the two-dimensional image, the coordinate information contained in the two-dimensional image, and the grayscale information of the two-dimensional image. It should be noted that the header files of the two-dimensional images corresponding to the same target part of the same human body contain the same sequence ID.
[0092] The position of the two-dimensional image in the image sequence can be expressed as the projection angle corresponding to the two-dimensional image in the three-dimensional image to be constructed.
[0093] According to the coordinate information of the two-dimensional image contained in the above header file, the projection angle of the two-dimensional image in the corresponding three-dimensional image can be determined.
[0094] Step S2022: Map each two-dimensional image into a three-dimensional image in sequence according to the position of the two-dimensional image in the image sequence.
[0095] According to the projection angles of each two-dimensional image in the corresponding three-dimensional image obtained by the above calculation, a corresponding three-dimensional image can be constructed.
[0096] Optionally, two-dimensional images corresponding to the same serial number are constructed to form a three-dimensional image. It can be understood that each three-dimensional image corresponds to multiple frames of two-dimensional images. In the embodiment of the present application, multiple frames of two-dimensional images are distinguished according to the ID of the two-dimensional image.
[0097] In this embodiment, a three-dimensional image is constructed based on the header data of multiple two-dimensional images. Since the three-dimensional image is constructed before the marked data of the terminal device is transmitted in this embodiment, the calculation process can be disassembled, the idle time of the AI device is utilized as much as possible, and the response time of the AI device is reduced.
[0098] Please refer to Figure 5 Optionally, the above step S204 determines the real size information of the part where the target two-dimensional coordinates are located according to the image selection information and the three-dimensional image corresponding to the target two-dimensional image, which can be obtained by the following steps S2041 to S2043.
[0099] Step S2041 : determining a three-dimensional image corresponding to the target two-dimensional image according to the identifier of the target two-dimensional image.
[0100] The identification of the target two-dimensional image may include but is not limited to: the ID of the target two-dimensional image and the sequence ID corresponding to the target two-dimensional image.
[0101] Optionally, the sequence ID corresponding to the target two-dimensional image is stored in the image archiving and communication device. When the user performs measurement on the terminal device, the image archiving and communication device determines the sequence ID corresponding to the target two-dimensional image according to the ID of the target two-dimensional image transmitted by the terminal device. Then, the target two-dimensional coordinates, the ID of the target two-dimensional image, and the sequence ID corresponding to the target two-dimensional image are sent to the AI device.
[0102] In this embodiment, the AI device can determine the 3D image corresponding to the target 2D image according to the sequence ID corresponding to the target 2D image. Since the 3D image is constructed by mapping one or more 2D image sequences, the AI device can also compare the ID of the target 2D image with the above 2D image to determine the position of the target 2D image in the image sequence.
[0103] Step S2042: determining the target three-dimensional coordinates of the target two-dimensional coordinates in the three-dimensional image according to the target two-dimensional coordinates and the three-dimensional image.
[0104] In the above embodiment, it can be understood that before the AI system receives the user operation data, i.e., the target two-dimensional coordinates, sent by the terminal device forwarded by the image archiving and communication device, it has already constructed the corresponding three-dimensional image based on the image sequence sent by the image archiving and communication device.
[0105] Afterwards, the AI system receives the target two-dimensional coordinates. In order to calculate the actual size information of the part where the target two-dimensional coordinates are located, the target two-dimensional coordinates need to be converted into a corresponding three-dimensional point or three-dimensional area in the constructed three-dimensional image. The specific conversion method will be described in detail in the following embodiments.
[0106] Step S2043, determining the real size information of the part where the target two-dimensional coordinates are located according to the target three-dimensional coordinates.
[0107] According to the conversion of the target two-dimensional coordinates into a corresponding three-dimensional point or three-dimensional area in the constructed three-dimensional image, the image segmentation algorithm and the coordinate system conversion algorithm can be used to obtain the real size information of the part where the target two-dimensional coordinates corresponding to the three-dimensional point or three-dimensional area are located.
[0108] Among them, the real size information can be interpreted as the actual size information of the boundary of the corresponding maximum cross-section of the target part in the human body or human tissue, the projection angle corresponding to the target two-dimensional image in the three-dimensional image, for example, it can include: the major axis and minor axis of the part, which is not limited in this application.
[0109] In this embodiment, the user selects a point or a frame of the target part on the image on the terminal device, and the AI device calculates the size of the part in the pre-established three-dimensional image based on the received coordinate information of the point or frame of the target part, and finally maps it into real size information. This method of calculating in the pre-established three-dimensional image saves the time required for the AI device to calculate the size information and speeds up the time to respond to user operations.
[0110] Optionally, see Figure 6 In the above step S2042, the target three-dimensional coordinates of the target two-dimensional coordinates in the three-dimensional image are determined according to the target two-dimensional coordinates and the three-dimensional image, which can be obtained by the following steps S301 to S302.
[0111] Step S301, determining the distance between the target two-dimensional coordinates and the origin of the spatial coordinate system where the three-dimensional image is located, and obtaining the spatial coordinate system distance.
[0112] After the AI device generates a three-dimensional image from a two-dimensional image sequence, the header information of the three-dimensional image may include: the coordinates of the origin of the image coordinate system where the three-dimensional image is located, and the image pixel resolution of the three-dimensional image.
[0113] Based on this, the absolute value of the difference between the two-dimensional coordinates of the target and the coordinates of the origin of the image coordinate system where the three-dimensional image is located is calculated to obtain the distance in the spatial coordinate system.
[0114] The above-mentioned spatial coordinate system distance can be interpreted as, from a computational perspective, the value required to convert the target two-dimensional coordinates into the same coordinate system as the three-dimensional image.
[0115] Step S302 , determining a ratio of a distance in the spatial coordinate system to an image pixel resolution, and obtaining target three-dimensional coordinates, wherein the image pixel resolution is a pixel resolution of a target three-dimensional image.
[0116] The target three-dimensional coordinates can be obtained by calculating the distance between the spatial coordinate system distance and the image pixel resolution of the three-dimensional image contained in the three-dimensional image header information. The target three-dimensional coordinates can be the three-dimensional coordinates corresponding to the input target two-dimensional coordinates in the image coordinate system of the three-dimensional image.
[0117] It can be understood that, since the three-dimensional image is constructed based on the two-dimensional image sequence, the image pixel resolution of the three-dimensional image is the same as the image pixel resolution included in the header file information of the two-dimensional image.
[0118] In this embodiment, by mapping the input target two-dimensional coordinates to a pre-constructed three-dimensional image, multiple sets of target two-dimensional coordinates generated by multiple operations of the client can be mapped to the image coordinate system of the same three-dimensional image for calculation, thereby simplifying the calculation process and improving the real-time performance of the method.
[0119] Alternatively, if Figure 7 As shown, in the above step S2043, the real size information of the part where the target two-dimensional coordinates are located is determined according to the target three-dimensional coordinates, which can be obtained by the following steps S401 to S402.
[0120] Step S401, determining the three-dimensional area corresponding to the clicked part in the three-dimensional image according to the target three-dimensional coordinates.
[0121] After the target three-dimensional coordinates are calculated from the target two-dimensional coordinates in the above embodiment, the image segmentation algorithm can be used to calculate the group of target two-dimensional coordinates. The specific image segmentation algorithm used is not limited in this application.
[0122] Step S402, determining the real size information of the part where the target two-dimensional coordinates are located according to the three-dimensional area corresponding to the clicked part in the three-dimensional image.
[0123] After converting the target two-dimensional coordinates into a corresponding three-dimensional point or three-dimensional area in the constructed three-dimensional image, the three-dimensional size information corresponding to the three-dimensional point or three-dimensional area can be obtained using an image segmentation algorithm. The three-dimensional size information can include: boundary information of the part corresponding to the three-dimensional point or three-dimensional area in the three-dimensional image, such as length and width. In addition, the projection angle corresponding to the two-dimensional image where the target two-dimensional image is located and the three-dimensional boundary information of the maximum cross section in the three-dimensional image can also be obtained.
[0124] According to the three-dimensional size information, a conversion algorithm can be used to calculate the real size information of the part where the target two-dimensional coordinates are located. The specific calculation method will be described in detail in the following embodiments.
[0125] In this embodiment, according to the target three-dimensional coordinates calculated in the above embodiment, a three-dimensional area is determined in the image coordinate system of the three-dimensional image, and then the corresponding real size information is determined according to the three-dimensional area. This method of completing the calculation in three-dimensional space and then converting it into real size ensures the accuracy of the calculation and the accessibility of two-dimensional data.
[0126] Alternatively, if Figure 8 As shown, in the above step S402, the real size information of the part where the target two-dimensional coordinates are located is determined according to the three-dimensional area corresponding to the clicked part in the three-dimensional image, which can be obtained by the following steps S4021 to S4025.
[0127] Step S4021, calculating the transposition of the three-dimensional region and the transposition matrix to obtain a transposed target part boundary matrix.
[0128] The transposed matrix is a matrix that can interchange the rows and columns of the matrix where the three-dimensional region is located to obtain a transposed target part boundary matrix, and the transposed matrix is determined by the matrix where the three-dimensional region is located.
[0129] Step S4022, calculating the product of the transposed target part boundary matrix and the diagonal matrix to obtain the transposed diagonal target part boundary matrix, wherein the diagonal matrix is obtained based on the image pixel resolution and the target two-dimensional coordinates.
[0130] The diagonal matrix may be a matrix in which all elements except the main diagonal line are 0. In this embodiment, the diagonal matrix is calculated based on the pixel resolution of the three-dimensional image and the target two-dimensional coordinates to obtain the most matching diagonal matrix.
[0131] Step S4023, multiplying the diagonally transposed target part boundary matrix with a preset general matrix to obtain a first target part boundary matrix.
[0132] Optionally, the preset general matrix may be a matrix having the same number of rows and columns as the diagonally transposed target part boundary matrix and having all 1 values, and after being multiplied by the diagonally transposed target part boundary matrix, a first target part boundary matrix is obtained.
[0133] It is understandable that the preset general matrix can be set according to actual needs, and this application does not limit it here.
[0134] Step S4024: Add the first target part boundary matrix to the preset general formula to obtain a second target part boundary matrix.
[0135] In this embodiment, the preset general formula value is 0. In actual use, the preset general formula value can be set as needed, and this application does not limit it here.
[0136] Step S4025, invert the boundary matrix of the second target part to obtain the real size information of the part where the target two-dimensional coordinates are located.
[0137] The calculation process from step S4021 to step S4025 can be obtained by the following formula:
[0138] (x, y, z)^T=D*S2*(i, j, k)^T+A
[0139] Among them, (x, y, z) represents the target two-dimensional coordinates corresponding to the real size information of the target part, that is, the calculation result. T is the transposition function, D is the preset general matrix, S2 is the diagonal matrix, (i, j, k) is used to represent the boundary coordinate value of the three-dimensional area. (i, j, k) can be used to represent a three-dimensional area, and A is the preset general formula.
[0140] It can be understood that if there are multiple boundary coordinate values for determining the three-dimensional region in the image, the corresponding target two-dimensional coordinates can be obtained by multiple calculations according to the above formula.
[0141] It should be noted that the image size in the two-dimensional image is equal to the real size information, so the real size information can be obtained by calculating the two-dimensional coordinates. In addition, the real size information represents the size information obtained at the projection angle of the two-dimensional image in the corresponding three-dimensional image.
[0142] In this embodiment, the calculated boundary coordinate values of the three-dimensional area are converted into corresponding two-dimensional coordinates to obtain real size information, thereby ensuring the accuracy of the calculation.
[0143] Optionally, the image detection method provided in the embodiment of the present application may further include: sending the target two-dimensional coordinates to an image archiving and communication device, and the image archiving and communication device saving the target two-dimensional coordinates.
[0144] The image archiving and communication device may include a storage function to store the two-dimensional coordinates of the target calculated by the AI device and then send them to the terminal device for display.
[0145] In this embodiment, the image archiving and communication device establishes a cache structure between the AI device and the terminal device, which not only ensures the efficiency of information interaction between the AI device and the terminal device, but also reduces the transmission pressure on the network. In addition, the image archiving and communication device saves the target two-dimensional coordinates, which is also convenient for data management and query.
[0146] like Fig. 9 As shown, this embodiment also provides an image detection device 10, which includes:
[0147] The receiving module 1001 is used to receive an image sequence sent by an image archiving and communication device, wherein the image sequence includes multiple frames of two-dimensional images, and the multiple frames of two-dimensional images include: an image displayed on a terminal device.
[0148] The receiving module 1001 is also used to receive image selection information sent by the image archiving and communication device, the image selection information including: the identifier of the target two-dimensional image selected by the user on the terminal device and the target two-dimensional coordinates clicked by the user on the target two-dimensional image.
[0149] A processing module 1002 is used to map multiple frames of two-dimensional images into three-dimensional images respectively;
[0150] The processing module 1002 is also used to determine the real size information of the part where the target two-dimensional coordinates are located according to the image selection information and the three-dimensional image corresponding to the target two-dimensional image;
[0151] The sending module 1003 sends the real size information of the part where the target two-dimensional coordinates are located to the terminal device via the image archiving and communication device.
[0152] As an optional implementation manner, the processing module 1002 is further configured to:
[0153] Determine a three-dimensional image corresponding to the target two-dimensional image according to the identifier of the target two-dimensional image;
[0154] Determine the target three-dimensional coordinates of the target two-dimensional coordinates in the three-dimensional image according to the target two-dimensional coordinates and the three-dimensional image;
[0155] According to the target's three-dimensional coordinates, determine the true size information of the part where the target's two-dimensional coordinates are located.
[0156] As an optional implementation manner, the processing module 1002 is further configured to:
[0157] Determine the distance between the target two-dimensional coordinates and the origin of the image coordinate system where the three-dimensional image is located, and obtain the distance in the spatial coordinate system;
[0158] The ratio of the spatial coordinate system distance to the image pixel resolution is determined to obtain the target three-dimensional coordinates, wherein the image pixel resolution is the pixel resolution of the target three-dimensional image.
[0159] As an optional implementation manner, the processing module 1002 is further configured to:
[0160] According to the target three-dimensional coordinates, determine the three-dimensional area corresponding to the clicked part in the three-dimensional image;
[0161] According to the three-dimensional area corresponding to the clicked part in the three-dimensional image, the real size information of the part where the target two-dimensional coordinates are located is determined.
[0162] As an optional implementation manner, the processing module 1002 is further configured to:
[0163] Calculate the transposition of the three-dimensional region and the transposed matrix to obtain the transposed target part boundary matrix;
[0164] Calculate the product of the transposed target part boundary matrix and the diagonal matrix to obtain the transposed diagonal target part boundary matrix, wherein the diagonal matrix is obtained based on the image pixel resolution and the target two-dimensional coordinates;
[0165] Multiplying the diagonally transposed target part boundary matrix with a preset general matrix to obtain a first target part boundary matrix;
[0166] Adding the first target part boundary matrix to the preset general formula to obtain a second target part boundary matrix;
[0167] The boundary matrix of the second target part is inverted to obtain the real size information of the part where the target two-dimensional coordinates are located.
[0168] As an optional implementation manner, the processing module 1002 is further configured to:
[0169] Analyze the header files of multiple frames of two-dimensional images respectively to obtain the position of each two-dimensional image in the image sequence;
[0170] According to the position of the two-dimensional image in the image sequence, each two-dimensional image is mapped into a three-dimensional image in sequence.
[0171] As an optional implementation, the image detection device 10 further includes:
[0172] The target two-dimensional coordinates are sent to the image archiving and communication device, and the image archiving and communication device saves the target two-dimensional coordinates.
[0173] This embodiment also provides an electronic device, such as Fig.10 As shown, the electronic device includes: a processor 501, a memory 502 and a bus, the memory 502 stores machine-readable instructions executable by the processor 501, when the electronic device is running, the processor 501 communicates with the memory through the bus, and the processor 501 executes the machine-readable instructions to perform the steps of the image detection method as in any of the aforementioned embodiments.
[0174] The memory 502, the processor 501 and the bus components are electrically connected to each other directly or indirectly to realize the transmission or interaction of data. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The image detection device includes at least one software function module that can be stored in the memory 502 or solidified in the operating system (operating system, OS) of the computer device in the form of software or firmware. The processor 501 is used to execute the executable modules stored in the memory 502, such as the software function modules and computer programs included in the image detection device.
[0175] Among them, the memory 502 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.
[0176] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. An image detection method, characterized in that: The method comprises: Receiving an image sequence sent by an image archiving and communication device, the image sequence including a plurality of two-dimensional image frames, the plurality of two-dimensional image frames including: an image displayed on a terminal device; Mapping the multiple frames of two-dimensional images into three-dimensional images respectively; Receiving image selection information sent by the image archiving and communication device, the image selection information including: an identifier of a target two-dimensional image selected by a user on the terminal device and a target two-dimensional coordinate clicked by the user on the target two-dimensional image; Determine the real size information of the part where the two-dimensional coordinates of the target are located according to the image selection information and the three-dimensional image corresponding to the two-dimensional image of the target; Sending the real size information of the part where the target two-dimensional coordinates are located to the image archiving and communication device for storage, and sending it to the terminal device via the image archiving and communication device; Mapping the multiple frames of two-dimensional images into three-dimensional images respectively includes: Parsing the header files of the multiple frames of two-dimensional images respectively to obtain the position of each of the two-dimensional images in the image sequence; Mapping each of the two-dimensional images into a three-dimensional image in sequence according to the position of the two-dimensional image in the image sequence; Determining the real size information of the part where the two-dimensional coordinates of the target are located according to the image selection information and the three-dimensional image corresponding to the two-dimensional image of the target includes: Determining a three-dimensional image corresponding to the target two-dimensional image according to the identifier of the target two-dimensional image; Determining the target three-dimensional coordinates of the target two-dimensional coordinates in the three-dimensional image according to the target two-dimensional coordinates and the three-dimensional image; Determine the real size information of the part where the two-dimensional coordinates of the target are located according to the three-dimensional coordinates of the target; Determining the real size information of the part where the two-dimensional coordinates of the target are located according to the three-dimensional coordinates of the target includes: Determining a three-dimensional area corresponding to the clicked position in the three-dimensional image according to the target three-dimensional coordinates; Determine the real size information of the part where the target two-dimensional coordinates are located according to the three-dimensional area corresponding to the clicked part in the three-dimensional image; The determining, according to the three-dimensional area corresponding to the clicked part in the three-dimensional image, the real size information of the part where the two-dimensional coordinates of the target are located, comprises: Calculating the transposition of the three-dimensional region and the transposition matrix to obtain a transposed target part boundary matrix; Calculating the product of the transposed target part boundary matrix and the diagonal matrix to obtain the transposed diagonal target part boundary matrix, wherein the diagonal matrix is obtained based on the image pixel resolution and the target two-dimensional coordinates; Multiplying the transposed diagonal target part boundary matrix with a preset general matrix to obtain a first target part boundary matrix; Adding the first target part boundary matrix to a preset general formula to obtain a second target part boundary matrix; The boundary matrix of the second target part is inverted to obtain the real size information of the part where the two-dimensional coordinates of the target are located.
2. The image detection method according to claim 1, characterized in that: The step of determining the target three-dimensional coordinates of the target two-dimensional coordinates in the three-dimensional image according to the target two-dimensional coordinates and the three-dimensional image includes: Determine the distance between the target two-dimensional coordinates and the origin of the image coordinate system where the three-dimensional image is located, and obtain the spatial coordinate system distance; The ratio of the spatial coordinate system distance to the image pixel resolution is determined to obtain the target three-dimensional coordinates, wherein the image pixel resolution is the pixel resolution of the target three-dimensional image.
3. The image detection method according to any one of claims 1 to 2, characterized in that: The method further comprises: The target two-dimensional coordinates are sent to an image archiving and communication device, and the image archiving and communication device stores the target two-dimensional coordinates.
4. An image detection device, characterized in that: The device comprises: A receiving module, used for receiving an image sequence sent by an image archiving and communication device, wherein the image sequence includes a plurality of two-dimensional images, and the plurality of two-dimensional images include: an image displayed on a terminal device; The receiving module is further used to receive image selection information sent by the image archiving and communication device, the image selection information including: an identifier of a target two-dimensional image selected by a user on the terminal device and a target two-dimensional coordinate clicked by the user on the target two-dimensional image; A processing module, used for mapping the multiple frames of two-dimensional images into three-dimensional images respectively; The processing module is also used to determine the real size information of the part where the two-dimensional coordinates of the target are located according to the image selection information and the three-dimensional image corresponding to the two-dimensional image of the target; A sending module, which sends the real size information of the part where the target two-dimensional coordinates are located to the terminal device via the image archiving and communication device; The processing module is specifically used for: Parsing the header files of the multiple frames of two-dimensional images respectively to obtain the position of each of the two-dimensional images in the image sequence; Mapping each of the two-dimensional images into a three-dimensional image in sequence according to the position of the two-dimensional image in the image sequence; The processing module is also specifically used for: Determining a three-dimensional image corresponding to the target two-dimensional image according to the identifier of the target two-dimensional image; Determining the target three-dimensional coordinates of the target two-dimensional coordinates in the three-dimensional image according to the target two-dimensional coordinates and the three-dimensional image; Determine the real size information of the part where the two-dimensional coordinates of the target are located according to the three-dimensional coordinates of the target; The processing module is also specifically used for: Determining a three-dimensional area corresponding to the clicked position in the three-dimensional image according to the target three-dimensional coordinates; Determine the real size information of the part where the target two-dimensional coordinates are located according to the three-dimensional area corresponding to the clicked part in the three-dimensional image; The processing module is also specifically used for: Calculating the transposition of the three-dimensional region and the transposition matrix to obtain a transposed target part boundary matrix; Calculating the product of the transposed target part boundary matrix and the diagonal matrix to obtain the transposed diagonal target part boundary matrix, wherein the diagonal matrix is obtained based on the image pixel resolution and the target two-dimensional coordinates; Multiplying the transposed diagonal target part boundary matrix with a preset general matrix to obtain a first target part boundary matrix; Adding the first target part boundary matrix to a preset general formula to obtain a second target part boundary matrix; The boundary matrix of the second target part is inverted to obtain the real size information of the part where the two-dimensional coordinates of the target are located.
5. An electronic device, characterized in that: The electronic device includes: a processor, a storage medium and a bus, the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the image detection method as described in any one of claims 1 to 3.
6. An image detection system, characterized in that: The system comprises: the electronic device, the image archiving and communication device and the terminal device as claimed in claim 5; The image archiving and communication device is communicatively connected with the electronic device and the terminal device respectively.
