Imaging method and system based on opto-acoustic-ultrasonic dual-mode device

Abstract

The invention is suitable for the technical field of image processing, and provides an imaging method and system based on a photoacoustic-ultrasonic dual-mode device, and the method comprises the steps: obtaining a photoacoustic imaging image and an ultrasonic imaging image through a mechanical part provided with the photoacoustic-ultrasonic dual-mode device in response to an imaging obtaining instruction; and based on the photoacoustic ultrasonic mapping relation corresponding to the photoacoustic ultrasonic dual-mode device, constructing the photoacoustic imaging image and a dual-mode imaging image corresponding to the ultrasonic imaging image. By adopting the method, image fusion can be performed on the imaging images constructed based on different coordinate systems through the photoacoustic ultrasonic mapping relation, so that imaging fusion errors caused by space deviation and direction difference between the coordinate systems in the bimodal fusion process can be reduced, and the use difficulty of a photoacoustic ultrasonic bimodal device is reduced; and the accuracy of the bimodal imaging image can be improved, so that the accuracy of subsequent automatic scanning, positioning and other applications can be improved.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to an imaging method and system based on a photoacoustic and ultrasonic dual-mode device. Background Technology

[0002] With the development of photoacoustic imaging (PAI) technology, laser-excited objects to generate ultrasonic signals, allowing for the simultaneous acquisition of both the object's light absorption characteristics and ultrasonic structural information, offering advantages such as deep penetration and high contrast. However, single-modal imaging often struggles to simultaneously achieve both image clarity and sensitivity. Combining ultrasound imaging (US) with photoacoustic imaging enables dual-modal imaging, providing richer information for medical diagnosis, surgical navigation, and precision targeted therapy.

[0003] In existing imaging technologies, photoacoustic-ultrasound dual-modal acquisition devices can be mounted on mechanical components, such as robotic arms or handheld positioning structures. These mechanical components are controlled to manage the acquisition of images by the photoacoustic-ultrasound dual-modal device, enabling automated scanning and precise positioning based on the dual-modal imaging images. However, due to spatial offsets and directional differences between the photoacoustic coordinate system, the ultrasonic imaging coordinate system, and the coordinate system corresponding to the mechanical components, the acquired images cannot be directly superimposed or registered. This increases the difficulty of using the photoacoustic-ultrasound dual-modal device, and the deviations between the coordinate systems can easily lead to result inconsistencies, thereby reducing the accuracy of the imaging image applications. Summary of the Invention

[0004] This application provides an imaging method and system based on a photoacoustic and ultrasonic dual-mode device, which can solve the problems of existing dual-mode imaging technology. When acquiring images through the photoacoustic and ultrasonic dual-mode device, there are spatial offsets and directional differences between the photoacoustic coordinate system, the ultrasonic imaging coordinate system, and the coordinate system corresponding to the mechanical components, resulting in the inability to directly overlay and register the acquired images, which leads to problems of high difficulty in use and low application accuracy.

[0005] In a first aspect, embodiments of this application provide an imaging method based on a photoacoustic and ultrasonic dual-mode device, including:

[0006] In response to an imaging acquisition command, a photoacoustic imaging image and an ultrasonic imaging image are acquired through a mechanical component equipped with a photoacoustic-ultrasound dual-mode device; the photoacoustic imaging image is acquired through a photoacoustic acquisition probe in the photoacoustic-ultrasound dual-mode device; the ultrasonic imaging image is acquired through an ultrasonic acquisition probe in the photoacoustic-ultrasound dual-mode device.

[0007] Based on the photoacoustic-ultrasound mapping relationship corresponding to the photoacoustic-ultrasound dual-mode device, the photoacoustic imaging image and the corresponding dual-mode imaging image of the ultrasound imaging image are constructed; the photoacoustic-ultrasound mapping relationship is determined by the mapping calibration system corresponding to the photoacoustic-ultrasound dual-mode device.

[0008] The mapping calibration system includes the photoacoustic-ultrasonic dual-mode device and an optical calibration module set in a preset layout space; the layout space includes at least one calibration line intersecting with a set of parallel lines.

[0009] In one possible implementation of the first aspect, acquiring a photoacoustic imaging image and an ultrasound imaging image via a photoacoustic-ultrasound dual-mode device in response to an imaging acquisition command includes:

[0010] The photoacoustic and ultrasonic dual-mode device acquires multiple sets of calibration images in the layout space; each set of calibration images includes photoacoustic calibration images and ultrasonic calibration images.

[0011] The first spatial coordinates corresponding to a preset imaging point in the photoacoustic calibration image and the second spatial coordinates corresponding to the imaging point in the ultrasonic calibration image are determined. The imaging point is the intersection between the shooting plane corresponding to the photoacoustic calibration image and the ultrasonic imaging image acquired by the photoacoustic-ultrasonic dual-mode device and the calibration line in the layout space. The first spatial coordinates are the spatial coordinates of the imaging point in the coordinate system corresponding to the optical acquisition probe. The second spatial coordinates are the spatial coordinates of the imaging point in the coordinate system corresponding to the ultrasonic acquisition probe.

[0012] The third spatial coordinates of the imaging point in the target coordinate system are determined based on the optical calibration module; the target coordinate system is a coordinate system constructed with the mechanical component where the photoacoustic-ultrasonic dual-mode device is located as a reference.

[0013] The photoacoustic ultrasound mapping relationship is generated based on the first spatial coordinates, the second spatial coordinates, and the third spatial coordinates corresponding to each set of calibration images.

[0014] In one possible implementation of the first aspect, the layout space includes at least two honeycomb panels disposed on the same base plate; the parallel line and the calibration line are straight lines between the two honeycomb panels.

[0015] Determining the third spatial coordinates of the imaging point under the optical calibration module based on the optical calibration module includes:

[0016] Based on the position of the honeycomb panel on the base plate, determine the first calibration coordinates of the first calibration point and the second calibration coordinates of the second calibration point on the two parallel lines respectively; the first calibration point is the projection point of the intersection of the first straight line in the set of parallel lines and the calibration line on the second straight line in the set of parallel lines; the second calibration point is the projection point of the intersection of the second straight line and the calibration line on the first straight line.

[0017] Based on the third and fourth calibration points between the imaging plane of the optical calibration module and the two parallel lines, the scaling factor corresponding to the imaging plane is determined; the third calibration point is the intersection point between the imaging plane and one straight line on the set of parallel lines; the fourth calibration point is the intersection point between the imaging plane and another straight line on the set of parallel lines; the scaling factor is:

[0018] k = |EF| / |EG|

[0019] Wherein, k is the scaling factor; F is the first pixel coordinate of the imaging point in the photoacoustic calibration image; E is the second pixel coordinate of the third calibration point in the photoacoustic calibration image; and G is the third pixel coordinate of the fourth calibration point in the photoacoustic calibration image.

[0020] Based on the scaling factor, the first calibration coordinates, and the second calibration coordinates, the fourth spatial coordinates of the imaging point in the coordinate system constructed based on the optical calibration module are determined;

[0021] The third spatial coordinates of the imaging point in the target coordinate system are determined based on the fourth spatial coordinates and the coordinate transformation matrix corresponding to the optical calibration module.

[0022] In one possible implementation of the first aspect, determining the fourth spatial coordinates of the imaging point under the optical calibration module based on the scaling factor, the first calibration coordinates, and the second calibration coordinates includes:

[0023] Based on the first calibration coordinates of the first calibration point on each coordinate axis, the second calibration coordinates of the second calibration point on each coordinate axis, and the scaling factor, the fourth spatial coordinates of the imaging point are determined; the fourth spatial coordinates are specifically identified as follows:

[0024]

[0025] in, P F x Let x be the fourth spatial coordinate of the imaging point on the x-axis; P A' xD' is the first calibration coordinate of the first calibration point on the x-axis; x The second calibration point is located at the second calibration coordinate on the x-axis. P F y Let be the fourth spatial coordinate of the imaging point on the y-axis; P A' y Let the first calibration point be the first calibration coordinate on the y-axis; P D' y The second calibration point is located at the second calibration coordinate on the y-axis. P F z A' represents the fourth spatial coordinate of the imaging point on the z-axis. z The first calibration point is located at the first calibration coordinate on the z-axis. P D' z The second calibration point has the second calibration coordinates on the z-axis.

[0026] In one possible implementation of the first aspect, the optical calibration module includes: an optical positioning instrument and a calibration tool; the optical positioning instrument is used to provide information about the world coordinate system;

[0027] Determining the third spatial coordinates of the imaging point in the target coordinate system based on the fourth spatial coordinates and the coordinate transformation matrix corresponding to the optical calibration module includes:

[0028] Based on the optical positioning instrument, a first transformation matrix is ​​determined to transform from the world coordinate system to the target coordinate system corresponding to the mechanical component where the photoacoustic-ultrasonic dual-mode device is located;

[0029] Based on the optical positioning instrument and the calibration tool, determine the second transformation matrix corresponding to the transformation from the world coordinate system to the target coordinate system;

[0030] Based on the first spatial coordinates, the first transformation matrix, and the second transformation matrix, the third spatial coordinates corresponding to the imaging point in the target coordinate system are determined; the third spatial coordinates are expressed as:

[0031]

[0032] in, S F is the third spatial coordinate of the imaging point in the target coordinate system; P F represents the fourth spatial coordinate; This is the first transformation matrix; Let be the second transformation matrix.

[0033] In one possible implementation of the first aspect, generating the photoacoustic-ultrasound mapping relationship based on the first spatial coordinates, the second spatial coordinates, and the third spatial coordinates corresponding to each set of calibration images includes:

[0034] For any set of calibration images corresponding to the first spatial coordinates, the second spatial coordinates, and the third spatial coordinates, a mapping relationship determination equation is constructed; the mapping relationship determination method is expressed as follows:

[0035]

[0036] in, PA F represents the first spatial coordinates; U F is the second spatial coordinate; S F represents the third spatial coordinates; This involves transforming the coordinate system corresponding to the photoacoustic acquisition probe to the first target matrix corresponding to the target coordinate system; This involves transforming the coordinate system corresponding to the ultrasonic acquisition probe to the second target matrix corresponding to the target coordinate system;

[0037] Based on the mapping relationship corresponding to each set of calibration images, an equation is determined to obtain the first target matrix and the second target matrix; the first target matrix is ​​represented as follows:

[0038]

[0039] in, PA F i The first spatial coordinates corresponding to the calibration image of the i-th group; S F i Let be the third spatial coordinates corresponding to the i-th group of calibration images; n is the total number of groups corresponding to the calibration images; argmin is the minimum value selection function; This is the first target matrix;

[0040] The second target matrix is ​​represented as:

[0041]

[0042] in, The second spatial coordinates are the coordinates corresponding to the calibration images in the i-th group; This is the second target matrix;

[0043] The photoacoustic-ultrasonic mapping relationship is obtained based on the first target matrix and the second target matrix.

[0044] In one possible implementation of the first aspect, the layout space includes N calibration lines intersecting a set of parallel lines; each set of calibration images includes N imaging points; and the layout space is located within a container filled with a preset medium.

[0045] Secondly, embodiments of this application provide an imaging calibration system based on a photoacoustic-ultrasound dual-mode device, comprising: a calibration device, a photoacoustic-ultrasound dual-mode device, an optical calibration module, and a layout space; the photoacoustic-ultrasound dual-mode device comprises: a photoacoustic acquisition probe and an ultrasonic acquisition probe.

[0046] The layout space includes at least one set of parallel lines and calibration lines intersecting with the set of parallel lines; the optical calibration module is disposed within the layout space;

[0047] The photoacoustic and ultrasonic dual-mode device is used to acquire multiple sets of calibration images in the layout space; each set of calibration images includes photoacoustic calibration images and ultrasonic calibration images.

[0048] The calibration device is used to: determine the first spatial coordinates corresponding to a preset imaging point in the photoacoustic calibration image, and determine the second spatial coordinates corresponding to the imaging point in the ultrasonic calibration image; the imaging point is the intersection between the shooting plane corresponding to the photoacoustic calibration image and the ultrasonic imaging image acquired by the photoacoustic and ultrasonic dual-mode devices and the calibration line in the layout space; the first spatial coordinates are the spatial coordinates of the imaging point in the coordinate system corresponding to the optical acquisition probe; the second spatial coordinates are the spatial coordinates of the imaging point in the coordinate system corresponding to the ultrasonic acquisition probe.

[0049] The third spatial coordinates of the imaging point in the target coordinate system are determined based on the optical calibration module; the target coordinate system is a coordinate system constructed with the mechanical component where the photoacoustic-ultrasonic dual-mode device is located as a reference.

[0050] Based on the first spatial coordinates, the second spatial coordinates, and the third spatial coordinates corresponding to each set of calibration images, a photoacoustic-ultrasound mapping relationship is generated for the photoacoustic-ultrasound dual-mode device; the photoacoustic-ultrasound mapping relationship is used to realize the coordinate transformation between the coordinate system corresponding to the photoacoustic acquisition probe and the coordinate system corresponding to the ultrasonic acquisition probe.

[0051] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any of the first aspects above.

[0052] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any of the first aspects above.

[0053] Fifthly, embodiments of this application provide a computer program product that, when run on a drone, causes the drone to perform the method described in any one of the first aspects above.

[0054] The beneficial effects of this application embodiment compared with the prior art are as follows: Before acquiring dual-modal imaging images using a photoacoustic-ultrasound dual-mode device, the photoacoustic-ultrasound mapping relationship corresponding to the photoacoustic-ultrasound dual-mode device configured on the mechanical component can be determined by a mapping calibration system. This photoacoustic-ultrasound mapping relationship enables coordinate system transformation. Then, in response to an imaging acquisition command, the aforementioned photoacoustic-ultrasound mapping relationship enables dual-modal imaging fusion between the photoacoustic and ultrasonic imaging images, achieving calibration and synthesis of the dual-modal images. Compared with existing imaging technologies, in this application embodiment, each imaging image does not use its own coordinate system. Instead, image fusion is performed on imaging images constructed based on different coordinate systems through the photoacoustic-ultrasound mapping relationship. This reduces imaging fusion errors caused by spatial offsets and directional differences between coordinate systems during dual-modal fusion, lowers the difficulty of using the photoacoustic-ultrasound dual-mode device, and improves the accuracy of dual-modal imaging images, thereby improving the accuracy of subsequent automated scanning and positioning applications. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the structure of an imaging system based on a photoacoustic and ultrasonic dual-mode device according to an embodiment of this application;

[0057] Figure 2 This is a schematic diagram illustrating the implementation of an imaging method based on a photoacoustic and ultrasonic dual-mode device provided in an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of a mapping and calibration scenario provided in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of the layout space provided in an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of the structure of a mapping calibration system provided in an embodiment of this application;

[0061] Figure 6 This is a flowchart illustrating the specific implementation of establishing a photoacoustic-ultrasound mapping relationship in an imaging method based on a photoacoustic-ultrasound dual-mode device, as provided in the second embodiment of this application.

[0062] Figure 7 This is a schematic diagram illustrating the determination of imaging points according to an embodiment of this application;

[0063] Figure 8 This is a schematic diagram of a calibration image provided in an embodiment of this application;

[0064] Figure 9 This is a schematic diagram illustrating the determination of photoacoustic-ultrasound mapping relationship according to an embodiment of this application;

[0065] Figure 10 This is a structural block diagram of an imaging device based on a photoacoustic and ultrasonic dual-mode device according to an embodiment of this application;

[0066] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0067] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0068] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0069] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0070] This application provides an imaging method and system based on a photoacoustic-ultrasound dual-mode device. This imaging method can be applied to an imaging system configured with a photoacoustic-ultrasound dual-mode device. For example, Figure 1 A schematic diagram of an imaging system based on a photoacoustic-ultrasound dual-mode device according to an embodiment of this application is shown. See also Figure 1 The imaging system based on a photoacoustic-ultrasound dual-mode device includes a processing device 11 and a mechanical component 12. The mechanical component 12 may have a photoacoustic-ultrasound dual-mode device 13 at its end. This device 13 has two imaging probes: a photoacoustic imaging probe 131 and an ultrasonic imaging probe 132. The photoacoustic imaging probe 131 can acquire photoacoustic images, and the ultrasonic imaging probe can acquire ultrasonic images. The mechanical component 12 can be manually controlled by the user or controlled by the processing device, for example, by controlling its movement or image acquisition. In response to a user's imaging acquisition command, the mechanical component 12 acquires ultrasonic and photoacoustic images through the photoacoustic-ultrasound dual-mode device 13 and sends them to the processing device 11. The processing device 11 uses locally stored photoacoustic-ultrasound mapping relationships to perform dual-modal fusion of the two images, resulting in a dual-modal imaging image.

[0071] Compared with imaging technology, each imaging image in this application embodiment does not use its own coordinate system. Instead, the imaging images constructed based on different coordinate systems are fused through photoacoustic-ultrasound mapping relationship. This can reduce the imaging fusion error caused by spatial offset and directional difference between coordinate systems during the dual-modal fusion process, reduce the difficulty of using the photoacoustic-ultrasound dual-mode device, and improve the accuracy of dual-modal imaging images. In turn, it can improve the accuracy of subsequent applications such as automated scanning and positioning.

[0072] Please see Figure 2 , Figure 2 This illustration shows a schematic diagram of an imaging method based on a photoacoustic-ultrasound dual-mode device according to an embodiment of this application. This imaging method is applied to the aforementioned processing device 11, meaning the executing entity in this embodiment can be the aforementioned processing device 11. Specifically, the processing device 11 is an electronic device, such as a computer, laptop, server, or smartphone. For ease of description, the executing entity will be described using an electronic device as an example. Specifically, the method includes the following steps:

[0073] In S201, in response to an imaging acquisition command, a photoacoustic imaging image and an ultrasonic imaging image are acquired through a mechanical component configured with a photoacoustic-ultrasound dual-mode device; the photoacoustic imaging image is acquired through a photoacoustic acquisition probe in the photoacoustic-ultrasound dual-mode device; and the ultrasonic imaging image is acquired through an ultrasonic acquisition probe in the photoacoustic-ultrasound dual-mode device.

[0074] In this embodiment, the electronic device can be electrically connected to the aforementioned mechanical component, and the movement of the mechanical component can be controlled by the electronic device. The mechanical component can be a robotic arm, a handheld positioning structure, or similar component. The mechanical component is equipped with a photoacoustic-ultrasound dual-mode device, which has two types of probes: a photoacoustic acquisition probe based on photoacoustic imaging, and an ultrasonic acquisition probe based on ultrasonic imaging.

[0075] In this embodiment, the user can directly control the movement of the aforementioned mechanical component and image acquisition, or remotely control the movement of the aforementioned mechanical component and image acquisition via an electronic device. Specifically, when the user needs to acquire dual-mode imaging images through the photoacoustic-ultrasound dual-mode device on the mechanical component, they can initiate an imaging acquisition command, such as clicking the corresponding shooting control or pressing the corresponding shooting button on the mechanical component. After receiving the aforementioned imaging acquisition command, the mechanical component can acquire at least two images through the two probes with different imaging principles on the photoacoustic-ultrasound dual-mode device: a photoacoustic imaging image acquired through the photoacoustic acquisition probe and an ultrasonic imaging image acquired through the ultrasonic acquisition probe.

[0076] In S202, based on the photoacoustic-ultrasound mapping relationship corresponding to the photoacoustic-ultrasound dual-mode device, the photoacoustic imaging image and the dual-mode imaging image corresponding to the ultrasound imaging image are constructed; the photoacoustic-ultrasound mapping relationship is determined by the mapping calibration system corresponding to the photoacoustic-ultrasound dual-mode device.

[0077] In this embodiment, the electronic device can store the photoacoustic-ultrasound mapping relationship corresponding to the photoacoustic-ultrasound dual-mode device. This mapping relationship is specifically used to transform the two imaging images to the same coordinate system, thereby aligning the pixels in the image and facilitating subsequent image fusion. Specifically, the photoacoustic imaging image is acquired by a photoacoustic acquisition probe; that is, the position of each pixel in the photoacoustic imaging image is determined based on the coordinate system corresponding to the photoacoustic acquisition probe. Similarly, the position of each pixel in the ultrasonic imaging image is also determined based on the coordinate system corresponding to the ultrasonic acquisition probe. Although both probes are mounted on mechanical components, since the photoacoustic acquisition probe and the ultrasonic acquisition probe are two different probes, there is a certain offset between their positions, resulting in a spatial offset between the two coordinate systems.

[0078] In this embodiment, in order to reduce errors caused by spatial offset, the electronic device can align the coordinate systems of the two imaging images through the above-mentioned photoacoustic-ultrasonic mapping relationship, that is, transform them to the same coordinate system, for example, transform them to the target coordinate system constructed based on the mechanical parts.

[0079] In some possible implementations, different components can correspond to different photoacoustic-ultrasonic mapping relationships. Electronic devices can obtain the corresponding photoacoustic-ultrasonic mapping relationship based on the component model of the mechanical component, thereby achieving the purpose of automatically obtaining the mapping relationship and improving the fusion efficiency of the imaging image.

[0080] In this embodiment, the electronic device can transform the photoacoustic imaging image and the ultrasound imaging image into the same coordinate system through the above-mentioned photoacoustic-ultrasound mapping relationship, and realize the image fusion of the two imaging images to generate a dual-modal imaging image. The electronic device can achieve precise positioning and navigation through the above-mentioned dual-modal imaging image, and can also be used in medical diagnosis, surgical navigation and precision targeted therapy and other application scenarios.

[0081] Optionally, the above image fusion method can be as follows: the electronic device performs coordinate transformation processing on the above photoacoustic imaging image through the photoacoustic-ultrasound mapping relationship to obtain a first transformed image, and performs coordinate transformation processing on the above ultrasound imaging image through the above photoacoustic-ultrasound mapping relationship to obtain a second transformed image. Since the above two imaging images have been transformed to the same coordinate system, the above first transformed image and second transformed image can be directly superimposed to obtain a dual-modal imaging image.

[0082] Optionally, the above image fusion method can also be as follows: the electronic device performs coordinate transformation processing on the above photoacoustic imaging image through the photoacoustic-ultrasound mapping relationship to obtain a first transformed image in the coordinate system corresponding to the ultrasonic acquisition probe, and performs image overlay processing on the first transformed image and the above ultrasonic imaging image to obtain a dual-modal imaging image; or, the electronic device can perform coordinate transformation processing on the above ultrasonic imaging image through the photoacoustic-ultrasound mapping relationship to obtain a second transformed image in the coordinate system corresponding to the photoacoustic acquisition probe, and performs image overlay processing on the second transformed image and the above photoacoustic imaging image to obtain a dual-modal imaging image.

[0083] As can be seen from the above, the imaging method based on a photoacoustic-ultrasound dual-mode device provided in this application embodiment allows the electronic device to determine the photoacoustic-ultrasound mapping relationship corresponding to the photoacoustic-ultrasound dual-mode device configured on the mechanical component through a mapping calibration system before acquiring dual-modal imaging images using the photoacoustic-ultrasound dual-mode device. This photoacoustic-ultrasound mapping relationship enables coordinate system transformation. Then, in response to an imaging acquisition command, the aforementioned photoacoustic-ultrasound mapping relationship enables dual-modal imaging fusion between the photoacoustic and ultrasonic imaging images, achieving calibration and synthesis of the dual-modal images. Compared with existing imaging technologies, in this application embodiment, each imaging image does not use its own coordinate system. Instead, image fusion is performed on imaging images constructed based on different coordinate systems through the photoacoustic-ultrasound mapping relationship. This reduces imaging fusion errors caused by spatial offsets and directional differences between coordinate systems during dual-modal fusion, lowers the difficulty of using the photoacoustic-ultrasound dual-mode device, and improves the accuracy of dual-modal imaging images, thereby improving the accuracy of subsequent automated scanning and positioning applications.

[0084] In this embodiment, to construct the photoacoustic-ultrasonic mapping relationship corresponding to the photoacoustic-ultrasonic dual-mode device on the aforementioned mechanical component, multiple sets of calibration images can be acquired in a mapping calibration scenario, and the aforementioned photoacoustic-ultrasonic mapping relationship can be established through these multiple sets of calibration images. For example, Figure 3 A schematic diagram of a mapping and calibration scenario provided in an embodiment of this application is shown. The mapping and calibration scenario includes a layout space 31 and an optical calibration module 32 disposed in the layout space. The optical calibration module 32 may include an optical calibration element 321 and an optical positioning device 322. The layout space 31 may include a base 311 and at least two honeycomb panels 312.

[0085] The aforementioned layout space is specifically used to establish a set of straight lines for determining the mapping relationship between different coordinates. These straight lines include at least three lines: a set of parallel lines that are mutually parallel, and a calibration line that intersects these parallel lines. For example, Figure 4 A schematic diagram of the layout space provided in one embodiment of this application is shown. See also Figure 4 As shown in (a), the two honeycomb panels have the same holes at the same position, so that two parallel straight lines, such as line 41 and line 42, can be set through the two sets of holes at the same position. By connecting the different holes, a calibration line that intersects the two parallel lines can be set in the layout space, namely line 43.

[0086] Optionally, the number of the above-mentioned line groups (i.e., lines 41 to 43) can be multiple. For example, see [link to example]. Figure 4As shown in (b), in addition to the line group 41 to 43 mentioned above, the layout space may also include two other line groups. The number of line groups can be set according to the actual situation. If each calibration image needs to contain N imaging points, N line groups can be set in the layout space. That is, the layout space can include N calibration lines, and each calibration line can correspond to the same or different parallel lines.

[0087] Based on the aforementioned mapping and calibration scenario, a mechanical component equipped with a photoacoustic-ultrasonic dual-mode device can be placed within the mapping and calibration scenario. The photoacoustic-ultrasonic dual-mode device then acquires the multiple sets of calibration images within the mapping and calibration scenario. For example, Figure 5 A schematic diagram of the structure of a mapping calibration system provided in one embodiment of this application is shown. See also Figure 5 As shown in (a), the mapping calibration system includes relevant components in the aforementioned mapping calibration scenario, such as a layout space and an optical calibration module. The optical calibration module includes an optical locator 51 and an optical calibration component 52. The optical calibration component 52 can be disposed on the layout space and the photoacoustic-ultrasonic dual-mode device 55, so that the optical locator can establish a mapping relationship between the world coordinate system and the target coordinate system (i.e., the coordinate system corresponding to the third spatial coordinates) through the optical calibration component deployed in the layout space and the optical calibration component deployed on the photoacoustic-ultrasonic dual-mode device 55, such as the subsequent second transformation matrix. The layout space can be disposed in a medium container 53, that is, the base plate and the honeycomb plate can be disposed within the medium container 53. At least one set of straight lines can be connected between the honeycomb plates. The set of straight lines can include the set of parallel lines and a calibration line intersecting the parallel lines. A medium capable of transmitting ultrasound and laser, such as water or gel, can be added to the medium container. The specific medium added to the medium container can be determined according to the actual situation.

[0088] The aforementioned mapping calibration system also includes a mechanical component 54, the end of which may be equipped with a photoacoustic-ultrasonic dual-mode device 55. The device's imaging direction can be towards the direction of the calibration line in the layout space to obtain a calibration image of the calibration line. The mechanical component can be fixedly installed in the mapping calibration system. Optionally, the mapping calibration system may also include a camera 56, which can acquire images in the layout space to assist the optical calibration module in establishing an optical calibration coordinate system.

[0089] In this embodiment, the optical positioning device 51, the photoacoustic and ultrasonic dual-mode device 55, and the camera 56 can be connected to the electronic device 57 to transmit the relevant acquisition data to the electronic device 57 so that the electronic device 57 can process the data to generate the photoacoustic and ultrasonic mapping relationship.

[0090] See Figure 5 As shown in (b), the electronic device can achieve transformations between multiple coordinate systems through the aforementioned mapping and calibration system. Specifically, the optical calibration module can achieve transformations between the coordinate system {P} corresponding to the optical calibration module and the world coordinate system {W}, i.e., through... Transformation matrix; it can also achieve the transformation between the world coordinate system {W} and the target coordinate system {S} through the optical calibration module, that is, through... Transformation matrix. By calibrating the image, points in the coordinate system {PA} corresponding to the photoacoustic acquisition probe can be transformed to the target coordinate system {S}, and points in the coordinate system {U} corresponding to the ultrasonic acquisition probe can also be transformed to the target coordinate system {S}, thereby realizing hand-eye spatial transformation.

[0091] Figure 6 This document illustrates a flowchart illustrating the specific implementation of establishing the photoacoustic-ultrasound mapping relationship in an imaging method based on a photoacoustic-ultrasound dual-mode device according to a second embodiment of this application. See also... Figure 6 As shown, relative to Figure 2 In the embodiment provided in this application, an imaging method based on a photoacoustic and ultrasonic dual-mode device may include steps S601 to S604 before step S201, as described in detail below:

[0092] In S601, multiple sets of calibration images are acquired in the layout space by the photoacoustic and ultrasonic dual-mode device; each set of calibration images includes photoacoustic calibration images and ultrasonic calibration images.

[0093] In this embodiment, at least one calibration line is configured within the layout space. The aforementioned photoacoustic and ultrasonic dual-mode device can align with the calibration line and acquire at least one set of calibration images containing the calibration line in the imaging frame. Each set of calibration images includes a photoacoustic calibration image acquired by the photoacoustic acquisition probe and an ultrasonic calibration image acquired by the ultrasonic acquisition probe.

[0094] In some implementations, the above-mentioned calibration image acquisition operation can be performed multiple times. Each time the image is acquired, the shooting angle of the photoacoustic-ultrasonic dual-mode device can be adjusted so that the imaging point between the imaging plane of the photoacoustic-ultrasonic dual-mode device and the calibration line is different each time the calibration image is acquired.

[0095] In some implementations, the electronic device can perform image processing on the received set of calibration images, such as image enhancement, to facilitate subsequent determination of spatial coordinates. Image enhancement may include sharpening, noise reduction, and color adjustment; the specific image enhancement method can be determined based on the actual situation and is not limited here.

[0096] In S602, the first spatial coordinates corresponding to the preset imaging point in the photoacoustic calibration image are determined, and the second spatial coordinates corresponding to the imaging point in the ultrasonic calibration image are determined; the imaging point is the intersection between the shooting plane corresponding to the photoacoustic calibration image and the ultrasonic imaging image acquired by the photoacoustic ultrasonic dual-mode device and the calibration line in the layout space; the first spatial coordinates are the spatial coordinates of the imaging point in the coordinate system corresponding to the optical acquisition probe; the second spatial coordinates are the spatial coordinates of the imaging point in the coordinate system corresponding to the ultrasonic acquisition probe.

[0097] In this embodiment, after receiving a set of calibration images from the photoacoustic-ultrasonic dual-mode device, the electronic device can determine the imaging points in the two calibration images respectively, and determine the spatial coordinates corresponding to the imaging points. For example, Figure 7 A schematic diagram illustrating the determination of imaging points according to an embodiment of this application is shown. See also... Figure 7 As shown, when the photoacoustic-ultrasonic dual-mode device acquires the calibration image, its corresponding imaging plane intersects with a set of parallel lines and calibration lines in the layout space. The parallel lines can be straight lines AB and CD. Points A', B', C', and D' are on the extension lines of points A, B, C, and D, respectively. Points A, B, C, and D are specifically points on the honeycomb plate. The spatial coordinates of points A, B, C, and D in the world coordinate system can be determined by an optical positioning instrument. The calibration line can be straight line B'C', and the intersection points of the calibration line and the honeycomb plate are points H and J, respectively.

[0098] The imaging plane corresponding to the above-mentioned photoacoustic and ultrasonic dual-mode device is plane P. The intersection points of plane P with the straight lines AB, CD and HJ in the layout space are points E, G and F, respectively. That is, point F is the intersection point between the calibration line and the above-mentioned imaging plane, which is the above-mentioned imaging point.

[0099] In some possible implementations, there are multiple sets of straight lines between the cellular panels, i.e., there are N calibration lines. The imaging points obtained above can be multiple, and the calibration lines of each set of straight lines will intersect with the imaging plane. The spatial coordinates of each imaging point can then be determined separately.

[0100] The photoacoustic calibration image is acquired by an optical acquisition probe, and the first spatial coordinates of the imaging point determined based on the photoacoustic calibration image are spatial coordinates in the coordinate system corresponding to the optical acquisition probe; the ultrasonic calibration image is acquired by an ultrasonic acquisition probe, and the second spatial coordinates of the imaging point determined based on the ultrasonic calibration image are spatial coordinates in the coordinate system corresponding to the ultrasonic acquisition probe.

[0101] For example, Figure 8 A schematic diagram of a calibration image provided in an embodiment of this application is shown. See also Figure 8 As shown in (a), image 81 is a photoacoustic calibration image obtained based on photoacoustic imaging, combined with... Figure 7 It can be determined that there are four sets of straight lines between the aforementioned honeycomb panels, each set intersecting the aforementioned imaging plane. That is, each set of lines has three intersection points. Specifically, the first set of lines (i.e., lines AB, CD, and HJ) intersects the imaging plane at points E, G, and F, respectively. The spatial coordinates of point F in image 81 are the first spatial coordinates of the aforementioned imaging point. (See also...) Figure 8 As shown in (b), image 82 is an ultrasound calibration image acquired based on ultrasound imaging, also combined with... Figure 7 In the ultrasound calibration images mentioned above, three intersection points with the first set of straight lines can also be acquired, namely point E, point G and point F. Among them, the spatial coordinates of point F in image 82 are the second spatial coordinates corresponding to the imaging point mentioned above.

[0102] In this embodiment, the electronic device can determine the first spatial coordinates of the imaging point under the photoacoustic calibration image based on the pixel size corresponding to the photoacoustic calibration image. For example, the coordinate system corresponding to the photoacoustic acquisition probe can be represented as {PA}, and correspondingly, the first spatial coordinates of the imaging point can be represented as: PA F.

[0103] In this embodiment, the electronic device can determine the first spatial coordinates of the imaging point under the ultrasound calibration image based on the pixel size corresponding to the ultrasound calibration image. For example, the coordinate system corresponding to the ultrasound acquisition probe can be represented as {U}, and correspondingly, the second spatial coordinates of the imaging point can be represented as: U F.

[0104] In S603, the third spatial coordinates of the imaging point in the target coordinate system are determined based on the optical calibration module; the target coordinate system is a coordinate system constructed with the mechanical component of the photoacoustic-ultrasonic dual-mode device as a reference.

[0105] In this embodiment, the electronic device can construct world coordinates using an optical calibrator in the optical calibration module. For example, world coordinates can be represented as {U}. Since the optical calibration module is located in the layout space, and the calibration lines are also fixedly located in the layout space, the relative positional relationship between the optical calibration module and the calibration lines is fixed. Therefore, the optical calibrator can accurately measure the world space coordinates of the imaging point in the world coordinate system, and based on the transformation relationship between the world coordinate system and the target coordinate system corresponding to the mechanical component, the world space coordinates are transformed to third space coordinates in the target coordinate system.

[0106] In one possible implementation, determining the third-space coordinates as described above may include the following steps:

[0107] In S603.1, based on the position of the honeycomb panel on the base plate, the first calibration coordinates of the first calibration point and the second calibration coordinates of the second calibration point on the two parallel lines are determined respectively.

[0108] In this embodiment, the first calibration point is the projection point of the intersection of the first straight line in the set of parallel lines and the calibration line onto the second straight line in the set of parallel lines. For example, the intersection of the parallel straight line AB and the calibration line HJ is point B', and the projection point of point B' onto the straight line CD is D'. The line connecting the intersection point B' and the projection point D' is perpendicular to both the straight line AB and the straight line CD. Similarly, the second calibration point is the projection point of the intersection of the second straight line in the set of parallel lines and the calibration point onto the first straight line. For example, the intersection of the parallel straight line CD and the calibration line HJ is point C', and the projection point of point C onto the straight line AB is A'. The line connecting the intersection point C' and the projection point A' is perpendicular to both the straight line AB and the straight line CD.

[0109] In this embodiment, the electronic device can use the optical calibrator in the optical calibration module to determine the first calibration coordinates of the first calibration point in the coordinate system corresponding to the optical positioning component in the optical calibration module, and to determine the second calibration coordinates of the second calibration point in the coordinate system corresponding to the optical calibration component in the optical calibration module.

[0110] Specifically, the method for determining the first calibration coordinate and the second calibration coordinate can be as follows:

[0111] The first calibration coordinate mentioned above can be expressed as:

[0112] P D' = [ P D x , P D y -|DD'|, P D z ]

[0113] in, P D' is the first calibration coordinate of the first calibration point mentioned above; P D x The coordinates of the first calibration point on the honeycomb plate corresponding to the connection point on the x-axis; P D y The coordinates of the first calibration point on the honeycomb panel on the y-axis are given. P D zLet |DD'| be the z-coordinate of the corresponding connection point on the honeycomb panel at the first calibration point; |DD'| is the distance from the first calibration point to the corresponding connection point on the honeycomb panel. Since the honeycomb panel is fixed to the floor of the layout space, and the optical calibration model is also fixed within the layout space, the coordinates of each hole on the honeycomb panel can be determined using optical calibration components and an optical calibration instrument. Therefore, the first calibration coordinates can be determined according to the above calculation formula. It should be noted that the coordinates of the corresponding connection point on the honeycomb panel are in the coordinate system {P} corresponding to the optical calibration model.

[0114] The second calibration coordinate mentioned above can be expressed as:

[0115] P A' = [ P A x , P A y +|DD'|, P A z ]

[0116] in, P A' is the second calibration coordinate of the second fixed point mentioned above; P A x The coordinates of the second calibration point on the corresponding connection point on the honeycomb plate on the x-axis; P A y The coordinates of the second calibration point on the cylinder plate corresponding to the connection point on the y-axis; P A z Let |DD'| be the z-coordinate of the corresponding connection point on the honeycomb plate of the second calibration point; |DD'| is the distance between the second calibration point and the corresponding connection point on the honeycomb plate.

[0117] In S603.2, based on the third and fourth calibration points between the imaging plane of the optical calibration module and the two parallel lines, the scaling factor corresponding to the imaging plane is determined; the scaling factor is:

[0118] k = |EF| / |EG|

[0119] Wherein, k is the scaling factor; F is the first pixel coordinate of the imaging point in the photoacoustic calibration image; E is the second pixel coordinate of the third calibration point in the photoacoustic calibration image; and G is the third pixel coordinate of the fourth calibration point in the photoacoustic calibration image.

[0120] In this embodiment, the third calibration point is the intersection of the imaging plane and a straight line on the set of parallel lines, such as... Figure 7 The point E where the straight line AB intersects the imaging plane P; the fourth calibration point is the intersection of the imaging plane and another straight line on the set of parallel lines, such as... Figure 7The point G where the straight line CD intersects the imaging plane P.

[0121] In some implementations, the second pixel coordinates can be based on the pixel coordinates of the third calibration point in the photoacoustic calibration image, and correspondingly, the third pixel coordinates can be based on the pixel coordinates of the fourth calibration point in the photoacoustic calibration image. Similarly, the first pixel coordinates can be based on the pixel coordinates of the imaging point in the photoacoustic calibration image.

[0122] In some implementations, the second pixel coordinates can be based on the pixel coordinates of the third calibration point in the ultrasound calibration image, and correspondingly, the third pixel coordinates can be based on the pixel coordinates of the fourth calibration point in the ultrasound calibration image. Similarly, the first pixel coordinates can be based on the pixel coordinates of the imaging point in the ultrasound calibration image.

[0123] In this embodiment, the calibration coordinates of the two calibration points are used to calculate the proportional relationship between line segments A'F and A'D, and this proportional relationship is the same as the ratio between line segments EF and EG. This proportional relationship is the same across different images and coordinate systems; therefore, the calibration coordinates corresponding to the two calibration points can be determined using a calibration image obtained in any coordinate system.

[0124] In S603.3, the fourth spatial coordinates of the imaging point in the coordinate system constructed based on the optical calibration module are determined according to the scaling factor, the first calibration coordinates, and the second calibration coordinates.

[0125] In this embodiment, the aforementioned fourth spatial coordinates are the spatial coordinates of the imaging point in the coordinate system corresponding to the optical calibration module. Specifically, the aforementioned fourth spatial coordinates can be expressed as:

[0126] The specific identifier for the fourth spatial coordinates is:

[0127]

[0128] in, P F x Let x be the fourth spatial coordinate of the imaging point on the x-axis; P A' x D' is the first calibration coordinate of the first calibration point on the x-axis; x The second calibration point is located at the second calibration coordinate on the x-axis. P F y Let be the fourth spatial coordinate of the imaging point on the y-axis; P A' y Let be the first calibration coordinate of the first calibration point on the y-axis; P D' yThe second calibration point is located at the second calibration coordinate on the y-axis. P F z A' represents the fourth spatial coordinate of the imaging point on the z-axis. z The first calibration point is located at the first calibration coordinate on the z-axis. P D' z The second calibration point has the second calibration coordinates on the z-axis.

[0129] In S603.4, the third spatial coordinates of the imaging point in the target coordinate system are determined based on the fourth spatial coordinates and the coordinate transformation matrix corresponding to the optical calibration module.

[0130] In this embodiment, the coordinate transformation matrix corresponding to the optical calibration module includes two matrices: a first transformation matrix for transforming the coordinate system {P} corresponding to the optical calibration module to the world coordinate system {W}, and a second transformation matrix for transforming the coordinate system {W} corresponding to the target coordinate system {S} corresponding to the mechanical component. Based on the above two matrices, the fourth spatial coordinates can be transformed to obtain the third spatial coordinates corresponding to the target coordinate system.

[0131] Specifically, the process of transforming from fourth spatial coordinates to third spatial coordinates can be described as follows:

[0132] Step 1: Based on the optical positioning instrument, determine the first transformation matrix for transforming from the world coordinate system to the target coordinate system corresponding to the mechanical component where the photoacoustic-ultrasonic dual-mode device is located.

[0133] In this embodiment, the optical calibration module includes an optical positioning device and a calibration tool; the optical positioning device is used to provide information about the world coordinate system. Since the optical calibration tool is fixedly mounted on the base plate of the aforementioned layout space, as... Figure 5 The optical calibration component 52 in the optical calibration module allows the transformation matrix between the coordinate system corresponding to the optical calibration module and the world coordinate system to be determined by measuring the calibration tool with an optical positioning instrument. This is the first transformation matrix mentioned above.

[0134] Step 2: Based on the optical positioning instrument and the calibration tool, determine the second transformation matrix corresponding to the transformation from the world coordinate system to the target coordinate system.

[0135] Step 3: Based on the first spatial coordinates, the first transformation matrix, and the second transformation matrix, determine the third spatial coordinates corresponding to the imaging point in the target coordinate system; the third spatial coordinates are expressed as:

[0136]

[0137] in, SF is the third spatial coordinate of the imaging point in the target coordinate system; P F represents the fourth spatial coordinate; This is the first transformation matrix; Let be the second transformation matrix.

[0138] In this embodiment, by constructing a first transformation matrix between the coordinate system corresponding to the optical calibration module and the world coordinate system, the fourth spatial coordinates can be transformed to the corresponding world coordinates in the world coordinate system. Then, by using a second transformation matrix between the world coordinate system and the target coordinate system corresponding to the mechanical component, the world coordinates corresponding to the imaging point can be transformed to the fourth coordinates in the target coordinate system. This enables the determination of the fourth spatial coordinates corresponding to the imaging in the mechanical component, thereby improving the accuracy of subsequent mapping relationship determination.

[0139] In S604, the photoacoustic ultrasound mapping relationship is generated based on the first spatial coordinates, the second spatial coordinates, and the third spatial coordinates corresponding to each set of calibration images.

[0140] In this embodiment, each set of calibration images can determine the third spatial coordinates of the imaging point in the target coordinate system, the first spatial coordinates in the coordinate system corresponding to the photoacoustic acquisition probe are determined by the photoacoustic calibration image, and the second spatial coordinates in the coordinate system corresponding to the ultrasonic acquisition probe are determined by the ultrasonic calibration image. By using the different coordinates of the same imaging point in the above three different coordinate systems, the transformation relationship between the above three coordinate systems can be determined, thereby obtaining the photoacoustic-ultrasonic mapping relationship transformed to the same coordinate system.

[0141] Optionally, the aforementioned photoacoustic-ultrasonic mapping relationship is specifically used to transform the spatial coordinates of each point to the coordinates in the target coordinate system corresponding to the mechanical component. The specific implementation process is as follows:

[0142] In S604.1, for any set of calibration images corresponding to the first spatial coordinates, the second spatial coordinates, and the third spatial coordinates, a mapping relationship determination equation is constructed; the mapping relationship determination method is expressed as follows:

[0143]

[0144] in, PA F represents the first spatial coordinates; U F is the second spatial coordinate; S F represents the third spatial coordinates; This involves transforming the coordinate system corresponding to the photoacoustic acquisition probe to the first target matrix corresponding to the target coordinate system; This is to transform the coordinate system corresponding to the ultrasonic acquisition probe to the second target matrix corresponding to the target coordinate system.

[0145] In this embodiment, the electronic device needs to determine the transformation relationship between the above three coordinate systems. Therefore, the corresponding transformation matrix can be used as the independent variable for solving. Thus, a first equation can be established to transform the coordinate system corresponding to the photoacoustic acquisition probe to the first target matrix corresponding to the target coordinate system, and a second equation can be established to transform the coordinate system corresponding to the ultrasonic acquisition probe to the second target matrix corresponding to the target coordinate system.

[0146] In some possible implementations, if a set of calibration images contains N imaging points, then a set of calibration images can establish a corresponding mapping relationship determination equation for each imaging point, that is, a set of calibration images can obtain N mapping relationship determination equations.

[0147] In S604.2, an equation is determined based on the mapping relationship corresponding to each group of calibration images to obtain the first target matrix and the second target matrix; the first target matrix is ​​represented as:

[0148]

[0149] in, PA F i The first spatial coordinates corresponding to the calibration image of the i-th group; S F i Let be the third spatial coordinates corresponding to the i-th group of calibration images; n is the total number of groups corresponding to the calibration images; argmin is the minimum value selection function; This is the first target matrix;

[0150] The second target matrix is ​​represented as:

[0151]

[0152] in, The second spatial coordinates are the coordinates corresponding to the calibration images in the i-th group; This is the second target matrix.

[0153] In this embodiment, the electronic device can transform the process of determining the above-mentioned photoacoustic-ultrasonic mapping relationship into a point-pair rigid registration process with a known pairing relationship. This calculation process can be solved using a closed-form solution method, such as using the Hom closed-form solution method to solve the above-mentioned multiple equations simultaneously, thereby obtaining the two target matrices corresponding to the target coordinate system.

[0154] In S604.3, the photoacoustic-ultrasonic mapping relationship is obtained based on the first target matrix and the second target matrix.

[0155] In the embodiments of this application, ...

[0156] For example, Figure 9 This illustration shows a schematic diagram of determining the photoacoustic-ultrasonic mapping relationship according to an embodiment of this application. Specifically, it may include the following steps:

[0157] In S91, a set of calibration images is acquired, including photoacoustic calibration images and ultrasonic calibration images.

[0158] In S92, image processing is performed on two images from a set of calibration images.

[0159] In S93, the imaging points contained in each set of calibration images are determined.

[0160] In S94, the spatial coordinates corresponding to the imaging point are determined by the optical calibration module, including the first spatial coordinate, the second spatial coordinate, and the third spatial coordinate.

[0161] In S95, the system of equations corresponding to this set of calibration images is constructed.

[0162] In S96, adjust the position of the calibration lines.

[0163] In S97, determine whether the calibration image acquisition for all groups has been completed. If not, proceed to S98; if yes, proceed to S99.

[0164] In S98, acquire the next set of calibration images.

[0165] In S99, solve all the equations simultaneously.

[0166] In S910, all systems of equations are solved.

[0167] In S911, the above-mentioned photoacoustic-ultrasonic mapping relationship is determined.

[0168] In this embodiment, Figure 10 This illustration shows a structural block diagram of an imaging device based on a photoacoustic-ultrasound dual-mode device according to an embodiment of this application. The imaging device includes units for performing various functions. Figure 2 The corresponding steps are implemented by the electronic device in the embodiment. Please refer to the details. Figure 2 and Figure 2 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown.

[0169] See Figure 10 An imaging device based on a photoacoustic-ultrasound dual-mode device includes:

[0170] The imaging acquisition unit 101 is used to acquire photoacoustic imaging images and ultrasonic imaging images through mechanical components configured with a photoacoustic-ultrasound dual-mode device in response to an imaging acquisition command; the photoacoustic imaging image is acquired through a photoacoustic acquisition probe in the photoacoustic-ultrasound dual-mode device; the ultrasonic imaging image is acquired through an ultrasonic acquisition probe in the photoacoustic-ultrasound dual-mode device.

[0171] The coordinate transformation unit 102 is used to construct the photoacoustic imaging image and the dual-modal imaging image corresponding to the ultrasonic imaging image based on the photoacoustic-ultrasonic mapping relationship corresponding to the photoacoustic-ultrasonic dual-mode device; the photoacoustic-ultrasonic mapping relationship is determined by the mapping calibration system corresponding to the photoacoustic-ultrasonic dual-mode device.

[0172] The mapping calibration system includes the photoacoustic-ultrasonic dual-mode device and an optical calibration module set in a preset layout space; the layout space includes at least one calibration line intersecting with a set of parallel lines.

[0173] It should be understood that, Figure 10 In the structural block diagram of the device shown, each module is used to perform... Figures 2 to 6 The steps in the corresponding embodiments, and for Figures 2 to 6 The steps in the corresponding embodiments have been explained in detail in the above embodiments. Please refer to them for details. Figures 2 to 6 as well as Figures 2 to 6 The relevant descriptions in the corresponding embodiments will not be repeated here.

[0174] Figure 11 This is a structural block diagram of an electronic device provided in another embodiment of this application. For example... Figure 11 The electronic device 1100 of this embodiment includes a processor 1110, a memory 1120, and a computer program 1130 stored in the memory 1120 and executable on the processor 1110, such as a program for a machine learning-based concrete self-compacting test method. When the processor 1110 executes the computer program 1130, it implements the steps in the various embodiments of the machine learning-based concrete self-compacting test method described above, for example... Figure 2 S201 to S202 are described above. Alternatively, the processor 1110 may implement the above when executing the computer program 1130. Figure 10 The functions of each module in the corresponding embodiments, for example, Figure 10 For details regarding the functions of units 101 to 102, please refer to [link / reference needed]. Figure 10 The relevant descriptions in the corresponding embodiments.

[0175] For example, computer program 1130 may be divided into one or more modules, one or more of which are stored in memory 1120 and executed by processor 1110 to complete this application. One or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 1130 in electronic device 1100. For example, computer program 1130 may be divided into various unit modules, each with the specific functions described above.

[0176] Electronic device 1100 may include, but is not limited to, processor 1110 and memory 1120. Those skilled in the art will understand that... Figure 11 This is merely an example of electronic device 1100 and does not constitute a limitation on electronic device 1100. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0177] The processor 1110 may be a central processing unit, or it may be other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0178] The memory 1120 can be an internal storage unit of the electronic device 1100, such as a hard disk or memory of the electronic device 1100. The memory 1120 can also be an external storage device of the electronic device 1100, such as a plug-in hard disk, smart memory card, flash memory card, etc. equipped on the electronic device 1100. Furthermore, the memory 1120 can include both internal storage units and external storage devices of the electronic device 1100.

[0179] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An imaging method based on a photoacoustic-ultrasound dual-mode device, characterized in that, include: In response to an imaging acquisition command, a photoacoustic imaging image and an ultrasonic imaging image are acquired through a mechanical component equipped with a photoacoustic-ultrasound dual-mode device; the photoacoustic imaging image is acquired through a photoacoustic acquisition probe in the photoacoustic-ultrasound dual-mode device; the ultrasonic imaging image is acquired through an ultrasonic acquisition probe in the photoacoustic-ultrasound dual-mode device. Based on the photoacoustic-ultrasound mapping relationship corresponding to the photoacoustic-ultrasound dual-mode device, the photoacoustic imaging image and the corresponding dual-mode imaging image of the ultrasound imaging image are constructed; the photoacoustic-ultrasound mapping relationship is determined by the mapping calibration system corresponding to the photoacoustic-ultrasound dual-mode device. The mapping calibration system includes the photoacoustic-ultrasonic dual-mode device and an optical calibration module set in a preset layout space; the layout space includes at least one calibration line intersecting with a set of parallel lines.

2. The imaging method according to claim 1, characterized in that, In response to an imaging acquisition command, the acquisition of photoacoustic and ultrasonic imaging images via a photoacoustic-ultrasound dual-mode device includes: The photoacoustic and ultrasonic dual-mode device acquires multiple sets of calibration images in the layout space; each set of calibration images includes photoacoustic calibration images and ultrasonic calibration images. The first spatial coordinates corresponding to a preset imaging point in the photoacoustic calibration image and the second spatial coordinates corresponding to the imaging point in the ultrasonic calibration image are determined. The imaging point is the intersection between the shooting plane corresponding to the photoacoustic calibration image and the ultrasonic imaging image acquired by the photoacoustic-ultrasonic dual-mode device and the calibration line in the layout space. The first spatial coordinates are the spatial coordinates of the imaging point in the coordinate system corresponding to the optical acquisition probe. The second spatial coordinates are the spatial coordinates of the imaging point in the coordinate system corresponding to the ultrasonic acquisition probe. The third spatial coordinates of the imaging point in the target coordinate system are determined based on the optical calibration module; the target coordinate system is a coordinate system constructed with the mechanical component where the photoacoustic-ultrasonic dual-mode device is located as a reference. The photoacoustic ultrasound mapping relationship is generated based on the first spatial coordinates, the second spatial coordinates, and the third spatial coordinates corresponding to each set of calibration images.

3. The imaging method according to claim 2, characterized in that, The layout space includes at least two honeycomb panels disposed on the same base plate; the parallel line and the calibration line are straight lines between the two honeycomb panels. Determining the third spatial coordinates of the imaging point under the optical calibration module based on the optical calibration module includes: Based on the position of the honeycomb panel on the base plate, determine the first calibration coordinates of the first calibration point and the second calibration coordinates of the second calibration point on the two parallel lines respectively; the first calibration point is the projection point of the intersection of the first straight line in the set of parallel lines and the calibration line on the second straight line in the set of parallel lines; the second calibration point is the projection point of the intersection of the second straight line and the calibration line on the first straight line. Based on the third and fourth calibration points between the imaging plane of the optical calibration module and the two parallel lines, the scaling factor corresponding to the imaging plane is determined; the third calibration point is the intersection point between the imaging plane and one straight line on the set of parallel lines; the fourth calibration point is the intersection point between the imaging plane and another straight line on the set of parallel lines; the scaling factor is: k = |EF| / |EG| Wherein, k is the scaling factor; F is the first pixel coordinate of the imaging point in the photoacoustic calibration image; E is the second pixel coordinate of the third calibration point in the photoacoustic calibration image; and G is the third pixel coordinate of the fourth calibration point in the photoacoustic calibration image. Based on the scaling factor, the first calibration coordinates, and the second calibration coordinates, the fourth spatial coordinates of the imaging point in the coordinate system constructed based on the optical calibration module are determined; The third spatial coordinates of the imaging point in the target coordinate system are determined based on the fourth spatial coordinates and the coordinate transformation matrix corresponding to the optical calibration module.

4. The imaging method according to claim 3, characterized in that, The step of determining the fourth spatial coordinates of the imaging point under the optical calibration module based on the scaling factor, the first calibration coordinates, and the second calibration coordinates includes: Based on the first calibration coordinates of the first calibration point on each coordinate axis, the second calibration coordinates of the second calibration point on each coordinate axis, and the scaling factor, the fourth spatial coordinates of the imaging point are determined; the fourth spatial coordinates are specifically identified as follows: Among them, PF x Let x be the fourth spatial coordinate of the imaging point on the x-axis; P A' x D' is the first calibration coordinate of the first calibration point on the x-axis; x The second calibration point is located at the second calibration coordinate on the x-axis. P F y Let be the fourth spatial coordinate of the imaging point on the y-axis; P A' y Let the first calibration point be the first calibration coordinate on the y-axis; P D' y The second calibration point is located at the second calibration coordinate on the y-axis. P F z A' represents the fourth spatial coordinate of the imaging point on the z-axis. z The first calibration point is located at the first calibration coordinate on the z-axis. P D' z The second calibration point has the second calibration coordinates on the z-axis.

5. The imaging method according to claim 3, characterized in that, The optical calibration module includes: an optical positioning instrument and calibration tools; the optical positioning instrument is used to provide information about the world coordinate system. Determining the third spatial coordinates of the imaging point in the target coordinate system based on the fourth spatial coordinates and the coordinate transformation matrix corresponding to the optical calibration module includes: Based on the optical positioning instrument, a first transformation matrix is ​​determined to transform from the world coordinate system to the target coordinate system corresponding to the mechanical component where the photoacoustic-ultrasonic dual-mode device is located; Based on the optical positioning instrument and the calibration tool, determine the second transformation matrix corresponding to the transformation from the world coordinate system to the target coordinate system; Based on the first spatial coordinates, the first transformation matrix, and the second transformation matrix, the third spatial coordinates corresponding to the imaging point in the target coordinate system are determined; the third spatial coordinates are expressed as: in, S F is the third spatial coordinate of the imaging point in the target coordinate system; P F represents the fourth spatial coordinate; This is the first transformation matrix; Let be the second transformation matrix.

6. The imaging method according to any one of claims 2-5, characterized in that, The step of generating the photoacoustic ultrasound mapping relationship based on the first spatial coordinates, the second spatial coordinates, and the third spatial coordinates corresponding to each group of calibrated images includes: For any set of calibration images corresponding to the first spatial coordinates, the second spatial coordinates, and the third spatial coordinates, a mapping relationship determination equation is constructed; the mapping relationship determination method is expressed as follows: in, PA F represents the first spatial coordinates; U F is the second spatial coordinate; S F represents the third spatial coordinates; This involves transforming the coordinate system corresponding to the photoacoustic acquisition probe to the first target matrix corresponding to the target coordinate system; This involves transforming the coordinate system corresponding to the ultrasonic acquisition probe to the second target matrix corresponding to the target coordinate system; Based on the mapping relationship corresponding to each set of calibration images, an equation is determined to obtain the first target matrix and the second target matrix; the first target matrix is ​​represented as follows: in, PA F i The first spatial coordinates corresponding to the calibration image of the i-th group; S F i Let be the third spatial coordinates corresponding to the i-th group of calibration images; n is the total number of groups corresponding to the calibration images; argmin is the minimum value selection function; This is the first target matrix; The second target matrix is ​​represented as: in, U F iv The second spatial coordinates are the coordinates corresponding to the calibration images in the i-th group; This is the second target matrix; The photoacoustic-ultrasonic mapping relationship is obtained based on the first target matrix and the second target matrix.

7. The imaging method according to any one of claims 2-6, characterized in that, The layout space includes N calibration lines that intersect with a set of parallel lines; each set of calibration images includes N imaging points; the layout space is located inside a container filled with a preset medium.

8. An imaging calibration system based on a photoacoustic-ultrasonic dual-mode device, characterized in that, include: Electronic equipment, photoacoustic and ultrasonic dual-mode device, optical calibration module and layout space; The photoacoustic and ultrasonic dual-mode device includes: a photoacoustic acquisition probe and an ultrasonic acquisition probe; The layout space includes at least one set of parallel lines and calibration lines intersecting with the set of parallel lines; the optical calibration module is disposed within the layout space; The photoacoustic and ultrasonic dual-mode device is used to acquire multiple sets of calibration images in the layout space; each set of calibration images includes photoacoustic calibration images and ultrasonic calibration images. The electronic device is used to: determine the first spatial coordinates corresponding to a preset imaging point in the photoacoustic calibration image, and determine the second spatial coordinates corresponding to the imaging point in the ultrasonic calibration image; the imaging point is the intersection between the shooting plane corresponding to the photoacoustic calibration image and the ultrasonic imaging image acquired by the photoacoustic ultrasonic dual-mode device and the calibration line in the layout space; the first spatial coordinates are the spatial coordinates of the imaging point in the coordinate system corresponding to the optical acquisition probe; the second spatial coordinates are the spatial coordinates of the imaging point in the coordinate system corresponding to the ultrasonic acquisition probe. The electronic device is used to: determine the third spatial coordinates of the imaging point in the target coordinate system based on the optical calibration module; the target coordinate system is a coordinate system constructed with the mechanical component of the photoacoustic-ultrasonic dual-mode device as a reference; The electronic device is used to: generate a photoacoustic-ultrasound mapping relationship corresponding to the photoacoustic-ultrasound dual-mode device based on the first spatial coordinates, the second spatial coordinates, and the third spatial coordinates corresponding to each set of calibration images; the photoacoustic-ultrasound mapping relationship is used to realize coordinate transformation between the coordinate system corresponding to the photoacoustic acquisition probe and the coordinate system corresponding to the ultrasonic acquisition probe.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images