Multi-modal motion compensation skin image display method and device, equipment and medium

Abstract

The embodiment of the invention discloses a multi-modal motion compensation skin image display method and device, equipment and a medium. A specific embodiment of the method comprises the following steps: performing probe space calibration processing on an ultrasonic probe and an optical coherence tomography probe to obtain a calibrated ultrasonic probe and a calibrated optical coherence tomography probe; determining skin scanning path information; generating a skin ultrasonic image set and a skin optical coherence tomography image set, performing gamma hybrid clustering to obtain a skin ultrasonic clustering image set and a skin optical coherence tomography clustering image set, and inputting the skin ultrasonic clustering image set and the skin optical coherence tomography clustering image set into a multi-modal image registration model to obtain a multi-modal registration skin image set; performing image block division processing on the multi-modal registration skin image set to obtain a multi-modal registration skin image block set; and performing Bayesian displacement compensation processing on the multi-modal registration skin image set to obtain a multi-modal motion compensation skin image set, and performing terminal image display. According to the embodiment, the quality of the skin image can be improved, and artifacts in the image are reduced.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a multimodal motion compensated skin image display method, device, equipment and medium. Background Art

[0002] The generation of probe-based skin ultrasound images and skin optical coherence tomography images has artifacts and blurring problems due to human breathing and slight shaking. How to accurately remove artifacts and blurring problems in ultrasound images and optical images has become a problem of increasing concern. For the display of motion-compensated images, the method usually adopted is: motion compensation is performed on a single skin ultrasound image or a single skin optical coherence tomography image to obtain a motion-compensated image, and the motion-compensated image is sent to the target terminal for image display.

[0003] However, it is found in practice that when the motion compensated image is displayed in the above-mentioned manner, the following technical problems often occur: Motion compensation is performed using a single type of image. Since the single type of image itself contains less skin feature information, the quality of the motion compensated image is lower. The motion compensated image may also have some artifacts and blurring problems that cannot be removed. More imaging images are required to ensure the quality of skin features, which increases the waste of transmission resources.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a multimodal motion compensated skin image display method, apparatus, device and medium to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a multimodal motion compensated skin image display method, comprising: performing probe space calibration processing on an ultrasound probe and an optical coherence tomography probe in a skin elasticity detection system to obtain a calibrated ultrasound probe and a calibrated optical coherence tomography probe; determining skin scanning path information of the calibrated ultrasound probe and the calibrated optical coherence tomography probe in a target user's skin to-be-detected area; based on the skin scanning path information, controlling the calibrated ultrasound probe and the calibrated optical coherence tomography probe to perform probe scanning imaging processing in the skin to-be-detected area to obtain a skin ultrasound image set and a skin optical coherence tomography image set; for each skin ultrasound image in the skin ultrasound image set and the Each skin optical coherence tomography image in the skin optical coherence tomography image set is subjected to gamma hybrid clustering processing to obtain a skin ultrasound clustering image set and a skin optical coherence tomography clustering image set; the above skin ultrasound clustering image set and the above skin optical coherence tomography clustering image set are input into a multimodal image registration model to obtain a multimodal registered skin image set; each multimodal registered skin image in the above multimodal registered skin image set is subjected to image block division processing to obtain a multimodal registered skin image block group set; based on the above multimodal registered skin image block group set, the above multimodal registered skin image set is subjected to Bayesian displacement compensation processing to obtain a multimodal motion compensated skin image set; and the above multimodal motion compensated skin image set is subjected to terminal image display.

[0008] In a second aspect, some embodiments of the present disclosure provide a multimodal motion compensated skin image display device, comprising: a probe space calibration unit, configured to perform probe space calibration processing on an ultrasound probe and an optical coherence tomography probe in a skin elasticity detection system, to obtain a calibrated ultrasound probe and a calibrated optical coherence tomography probe; a determination unit, configured to determine skin scanning path information of the calibrated ultrasound probe and the calibrated optical coherence tomography probe in a skin to-be-detected area of ​​a target user; a control unit, configured to control the calibrated ultrasound probe and the calibrated optical coherence tomography probe to perform probe scanning imaging processing in the skin to-be-detected area according to the skin scanning path information, to obtain a skin ultrasound image set and a skin optical coherence tomography image set; a gamma hybrid clustering unit, configured to perform probe space calibration processing on each skin ultrasound image in the skin ultrasound image set and the skin optical coherence tomography image set; Each skin optical coherence tomography image in the skin optical coherence tomography image set is subjected to gamma hybrid clustering processing to obtain a skin ultrasound clustering image set and a skin optical coherence tomography clustering image set; an input unit is configured to input the above-mentioned skin ultrasound clustering image set and the above-mentioned skin optical coherence tomography clustering image set into a multimodal image registration model to obtain a multimodal registered skin image set; an image block division unit is configured to perform image block division processing on each multimodal registered skin image in the above-mentioned multimodal registered skin image set to obtain a multimodal registered skin image block group set; a Bayesian displacement compensation unit is configured to perform Bayesian displacement compensation processing on the above-mentioned multimodal registered skin image set according to the above-mentioned multimodal registered skin image block group set to obtain a multimodal motion compensated skin image set; an image display unit is configured to perform terminal image display on the above-mentioned multimodal motion compensated skin image set.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0011] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the multimodal motion-compensated skin image display method of some embodiments of the present disclosure can improve the quality of the skin image and reduce the artifacts in the image caused by the human body's micro-breathing through the registration and motion compensation of the multimodal images. Specifically, the quality of the related motion-compensated image is lower, and the motion-compensated image may still have some artifacts and blurring problems that cannot be removed. More imaging images are required under the condition of ensuring the quality of skin features, which increases the waste of transmission resources. The reason is that: motion compensation is performed by a single type of image, and the single type of image itself contains less skin feature information, resulting in lower quality of the motion-compensated image, and the motion-compensated image may still have some artifacts and blurring problems that cannot be removed. More imaging images are required under the condition of ensuring the quality of skin features, which increases the waste of transmission resources. Based on this, the multimodal motion-compensated skin image display method of some embodiments of the present disclosure can first perform probe space calibration processing on the ultrasonic probe and the optical coherence tomography probe in the skin elasticity detection system to obtain a calibrated ultrasonic probe and a calibrated optical coherence tomography probe. Here, the coordinate systems corresponding to the ultrasonic probe and the optical coherence tomography probe are converted to the coordinate system of the target user, the positioning accuracy of the probe is improved, and the position information and direction information of the probe are integrated into the imaging image, thereby improving the accuracy of subsequent imaging positioning scanning. Secondly, the skin scanning path information of the calibrated ultrasonic probe and the calibrated optical coherence tomography probe in the target user's skin to be detected area is determined. Here, a more accurate probe scanning path can be determined, which is convenient for subsequent control of the scanning of the optical coherence tomography probe and the ultrasonic probe, and reduces the overlap between the generated images. Thirdly, according to the skin scanning path information, the calibrated ultrasonic probe and the calibrated optical coherence tomography probe are controlled to perform probe scanning imaging processing in the skin to be detected area to obtain a skin ultrasonic image set and a skin optical coherence tomography image set. Here, the overlap of skin feature information between the skin ultrasonic image in the generated skin ultrasonic image set and the skin optical coherence tomography image in the skin optical coherence tomography image set can be reduced, and redundant information and data volume can be reduced, so as to improve the efficiency of subsequent registration and motion compensation. Subsequently, gamma hybrid clustering processing is performed on each skin ultrasound image in the skin ultrasound image set and each skin optical coherence tomography image in the skin optical coherence tomography image set to obtain a skin ultrasound cluster image set and a skin optical coherence tomography cluster image set. Here, the detailed information of different skin tissue structures in the skin ultrasound image and the skin optical coherence tomography image can be accurately identified, and the determination of the position correspondence relationship of the subsequent image registration is facilitated by considering the probability distribution of the gray value. Then, the skin ultrasound cluster image set and the skin optical coherence tomography cluster image set are input into the multimodal image registration model to obtain a multimodal registered skin image set.Here, the characteristic information of skin ultrasound clustering images and skin optical coherence tomography clustering images can be automatically learned through the multimodal image registration model to improve the accuracy of multimodal image registration. Afterwards, each multimodal registered skin image in the above multimodal registered skin image set is subjected to image block division processing to obtain a multimodal registered skin image block set. Here, the multimodal registered skin image block set is convenient for subsequent motion compensation based on image blocks to improve the quality of the image. Then, according to the above multimodal registered skin image block set, the above multimodal registered skin image set is subjected to Bayesian displacement compensation processing to obtain a multimodal motion compensated skin image set. Here, the artifact problem in the multimodal registered image can be effectively removed, the influence of noise on the image can be reduced, and the quality of the multimodal motion compensated skin image can be improved. Finally, the terminal image display of the above multimodal motion compensated skin image set is performed. Here, since the multimodal motion compensated skin image contains more skin detail information and tissue structure information, the number of skin images can be reduced under the same skin feature information, thereby reducing the waste of transmission resources. It can be concluded that the multimodal motion compensated skin image display method can improve the quality of skin images and reduce artifacts in the image caused by human body's micro-breathing through multimodal image registration and motion compensation. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flow chart of some embodiments of a multimodal motion compensated skin image display method according to the present disclosure;

[0014] Figure 2 is a schematic structural diagram of some embodiments of a multi-modal motion-compensated skin image display device according to the present disclosure;

[0015] Figure 3 is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure;

[0016] Figure 4 is a scene diagram of a skin elasticity detection system in some embodiments of the multimodal motion compensated skin image display method according to the present disclosure;

[0017] Figure 5 It is a schematic diagram of skin ultrasound clustering images at different time sequences obtained by performing gamma hybrid clustering processing on skin ultrasound images in some embodiments of the multimodal motion compensated skin image display method disclosed in the present invention. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] Figure 1 The process 100 of some embodiments of the multimodal motion compensated skin image display method according to the present disclosure is shown. The multimodal motion compensated skin image display method comprises the following steps:

[0025] Step 101, performing probe space calibration processing on an ultrasound probe and an optical coherence tomography probe in a skin elasticity detection system to obtain a calibrated ultrasound probe and a calibrated optical coherence tomography probe.

[0026] In some embodiments, the execution subject (e.g., electronic device) of the multimodal motion compensated skin image display method can obtain probe space calibration processing of the ultrasound probe and the optical coherence tomography probe in the skin elasticity detection system through a wired connection or a wireless connection to obtain a calibrated ultrasound probe and a calibrated optical coherence tomography probe. The skin elasticity detection system can be a detection system for detecting elasticity information of the target user's skin. Figure 4 As shown. The skin elasticity detection system can be composed of a mechanical arm, an optical imaging device, an ultrasonic imaging device and a display device. The mechanical arm can include an ultrasonic probe and an optical coherence tomography probe. The mechanical arm can be a device that controls the optical coherence tomography probe and the ultrasonic probe to scan and image the ex vivo skin or the skin of the target user through the optical imaging device and the ultrasonic imaging device respectively. The display device is a display screen for terminal display of the generated image. The ultrasonic probe can be a probe for sending ultrasonic waves and receiving reflected ultrasonic waves. The optical coherence tomography probe can be a probe that obtains information about the reflected and scattered light waves of the internal structure of the target user's skin by emitting near-infrared light waves. The calibrated ultrasonic probe and the calibrated optical coherence tomography probe can be probes that convert the ultrasonic probe and the optical coherence tomography probe into the three-dimensional space where the target user is located.

[0027] In some optional implementations of some embodiments, performing probe space calibration processing on an ultrasound probe and an optical coherence tomography probe in a skin elasticity detection system to obtain a calibrated ultrasound probe and a calibrated optical coherence tomography probe may include the following steps:

[0028] The first step is to obtain an ultrasonic calibration skin image corresponding to the ultrasonic probe and an optical coherence tomography calibration skin image corresponding to the optical coherence tomography probe. The ultrasonic calibration skin image may be a skin image used to calibrate the ultrasonic probe. The optical coherence tomography calibration skin image may be a skin image used to calibrate the optical coherence tomography probe.

[0029] In the second step, corner point extraction is performed on the ultrasonically calibrated skin image and the optical coherence tomography calibrated skin image to obtain an ultrasonic skin corner point information set and an optical skin corner point information set. The ultrasonic skin corner point information in the ultrasonic skin corner point information set may be information of pixel points whose pixel values ​​change drastically in the ultrasonically calibrated skin image. The optical skin corner point information in the optical skin corner point information set may be information of pixel points whose pixel values ​​change drastically in the optical coherence tomography calibrated skin image. The corner point extraction may be corner point extraction performed using a corner point detection algorithm. The corner point detection algorithm may be a Harris corner point detection algorithm.

[0030] The third step is to determine the three-dimensional ultrasonic coordinate information of each ultrasonic skin corner information in the above-mentioned skin to-be-detected area in the above-mentioned ultrasonic skin corner information set to obtain a three-dimensional ultrasonic coordinate information set. The three-dimensional ultrasonic coordinate information in the above-mentioned three-dimensional ultrasonic coordinate information set may be the three-dimensional coordinate information of the above-mentioned ultrasonic skin corner information in the Cartesian coordinate system corresponding to the target user.

[0031] The fourth step is to determine the three-dimensional optical coordinate information of each optical skin corner information in the optical skin corner information set in the skin to-be-detected area to obtain a three-dimensional optical coordinate information set. The three-dimensional optical coordinate information in the three-dimensional optical coordinate information set may be the three-dimensional coordinate information of the optical skin corner information in the Cartesian coordinate system corresponding to the target user.

[0032] The fifth step is to perform spatial transformation on each ultrasonic skin corner point information in the ultrasonic skin corner point information set and the three-dimensional ultrasonic coordinate information corresponding to the three-dimensional ultrasonic coordinate information set to obtain an ultrasonic spatial transformation matrix. The ultrasonic spatial transformation matrix can represent a transformation matrix that transforms the coordinate system where the ultrasonic probe is located into the three-dimensional spatial coordinate system where the target user is located.

[0033] The sixth step is to perform spatial transformation on each optical skin corner information in the optical skin corner information set and the three-dimensional space ultrasound coordinate information corresponding to the three-dimensional space optical coordinate information set to obtain an optical space transformation matrix. The optical space transformation matrix can represent a transformation matrix that transforms the coordinate system where the optical coherence tomography probe is located into the three-dimensional space coordinate system where the target user is located.

[0034] The seventh step is to perform spatial calibration processing on the ultrasonic probe and the optical coherence tomography probe according to the ultrasonic space conversion matrix and the optical space conversion matrix to obtain a calibrated ultrasonic probe and a calibrated optical coherence tomography probe.

[0035] As an example, the execution entity may input the ultrasonic skin corner information set into the ultrasonic space conversion matrix, and input the optical skin corner information set into the optical space conversion matrix to obtain a calibrated ultrasonic probe and a calibrated optical coherence tomography probe.

[0036] Step 102: Determine skin scanning path information of the calibrated ultrasound probe and the calibrated optical coherence tomography probe in the skin area to be detected of the target user.

[0037] In some embodiments, the execution subject may determine the skin scanning path information of the calibrated ultrasound probe and the calibrated optical coherence tomography probe in the target user's skin detection area. The target user may be a user who needs to undergo skin detection. The skin detection area may be a skin area of ​​the target user waiting for ultrasound detection and optical coherence tomography detection. The skin scanning path information may be the information of the path of the calibrated ultrasound probe and the calibrated optical coherence tomography probe in the skin detection area. In practice, the execution subject may first set the probe scanning target point. The target point may be a skin area where a lesion occurs in the detection area. For example, the target point may be a skin area with spots. Again, using the path generation algorithm, the initial body surface needle entry point is set by puncturing the target point, and a projection section of the ultrasound probe in the two-dimensional image is generated. The projection section may be a two-dimensional image of the skin tissue structure generated by the calibrated ultrasound probe and the calibrated optical coherence tomography probe receiving the reflected signal at a preset angle position. Then, switch the two-dimensional image, observe the relationship between the projected section and the surrounding tissue, adjust the needle entry point on the body surface until the projected section does not pass through any important anatomical structure, generate a simulated puncture needle track, determine the distance from the needle track to the surrounding tissue, and if the distance is less than or equal to the preset distance threshold, complete the puncture path planning. Among them, the switching of the two-dimensional image can be the projection section of the two-dimensional image generated after adjusting the position and angle of the calibrated ultrasound probe and the calibrated optical coherence tomography probe. The above-mentioned preset distance threshold can be the maximum value of the pre-set distance.

[0038] Step 103, according to the skin scanning path information, control the calibrated ultrasound probe and the calibrated optical coherence tomography probe to perform probe scanning imaging processing in the skin to-be-detected area to obtain a skin ultrasound image set and a skin optical coherence tomography image set.

[0039] In some embodiments, the execution subject can control the calibrated ultrasound probe and the calibrated optical coherence tomography probe to perform probe scanning imaging processing on the skin to-be-detected area according to the skin scanning path information to obtain a skin ultrasound image set and a skin optical coherence tomography image set. The skin ultrasound image in the skin ultrasound image set can be a skin image with a deeper detection depth obtained by the calibrated ultrasound probe receiving ultrasound reflected from the skin area to be detected. The skin optical coherence tomography image in the skin optical coherence tomography image set can be a skin image with a better resolution obtained by the calibrated optical coherence tomography probe receiving near-infrared light reflected light signals.

[0040] As an example, the above-mentioned execution entity can control the above-mentioned calibrated ultrasound probe and the above-mentioned calibrated optical coherence tomography probe to perform probe scanning and imaging processing in the above-mentioned skin to-be-detected area according to the skin scanning path corresponding to the above-mentioned skin scanning path information, and obtain a skin ultrasound image set and a skin optical coherence tomography image set.

[0041] Step 104, performing gamma hybrid clustering processing on each skin ultrasound image in the skin ultrasound image set and each skin optical coherence tomography image in the skin optical coherence tomography image set to obtain a skin ultrasound cluster image set and a skin optical coherence tomography cluster image set.

[0042] In some embodiments, the execution entity may perform gamma hybrid clustering processing on each skin ultrasound image in the skin ultrasound image set and each skin optical coherence tomography image in the skin optical coherence tomography image set to obtain a skin ultrasound cluster image set and a skin optical coherence tomography cluster image set. The skin ultrasound cluster images in the skin ultrasound cluster image set may be images obtained by clustering and blocking each ultrasound pixel point in the skin ultrasound image. The skin optical coherence tomography cluster images in the skin optical coherence tomography cluster image set may be images obtained by clustering and blocking each optical pixel point in the skin optical coherence tomography cluster image. Figure 5 The figure shows a schematic diagram of skin ultrasound clustering images at different time sequences obtained by performing gamma hybrid clustering processing on skin ultrasound images.

[0043] In some optional implementations of some embodiments, the above-mentioned performing gamma hybrid clustering processing on each skin ultrasound image in the above-mentioned skin ultrasound image set and each skin optical coherence tomography image in the above-mentioned skin optical coherence tomography image set to obtain a skin ultrasound cluster image set and a skin optical coherence tomography cluster image set may include the following steps:

[0044] In the first step, based on the above skin ultrasound image set, the following first input step is performed:

[0045] Sub-step 1, determining an ultrasound clustering value for the skin ultrasound image set and an optical clustering value for the skin optical coherence tomography image set. The ultrasound clustering value may be the number of clusters for the skin ultrasound image set. The optical clustering value may be the number of non-clustered clusters for the skin optical coherence tomography image set. The determination may be made using an elbow algorithm.

[0046] Sub-step 2: performing image clustering processing on the skin ultrasound image set according to the ultrasound clustering value to obtain a skin ultrasound image cluster set. The skin ultrasound image cluster in the skin ultrasound image cluster set may be a cluster consisting of skin ultrasound images having the smallest distance from each skin ultrasound image to the cluster.

[0047] As an example, the execution subject may utilize a clustering algorithm to perform image clustering processing on the skin ultrasound image set according to the ultrasound clustering value to obtain a skin ultrasound image cluster set.

[0048] Sub-step 3, based on each skin ultrasound image cluster in the above skin ultrasound image cluster set, perform the following optimization steps:

[0049] In a first sub-step, a ratio of the number of skin ultrasound images included in the skin ultrasound image cluster to the number of skin ultrasound images is determined as an initial gamma mixing weight value, wherein the number of skin ultrasound images is the number of skin ultrasound images included in the skin ultrasound image set.

[0050] The second sub-step is to determine the pixel mean of each skin ultrasound image included in the skin ultrasound image cluster as the initial gamma mixing mean.

[0051] The third sub-step is to determine the square root of the mean of the sum of squares of the differences between the pixel values ​​of each skin ultrasound image and the pixel mean as the initial gamma mixing standard deviation.

[0052] In a fourth sub-step, the initial gamma mixing weight value, the initial gamma mixing mean value and the initial gamma mixing standard deviation are determined as a mixing model parameter set of the initial gamma mixing model for the skin ultrasound image cluster. The initial gamma mixing model may be a probability model for clustering by probability distribution of grayscale values ​​of each pixel of the skin ultrasound image.

[0053] The fifth sub-step is to optimize the initial gamma mixing model according to the mixing model parameter set to obtain an optimized gamma mixing model, wherein the optimized gamma mixing model can be a gamma mixing model with the optimal gamma mixing weight value, gamma mixing mean and gamma mixing standard deviation.

[0054] As an example, the execution subject may utilize the maximum expectation algorithm to optimize the initial gamma mixture model according to the mixture model parameter set to obtain an optimized gamma mixture model.

[0055] Sub-step 4: input the above skin ultrasound image clusters into the above optimized gamma mixture model to obtain a skin ultrasound cluster image set.

[0056] In the second step, the skin optical coherence tomography image set and the optical clustering value are determined as the skin ultrasound image set and the ultrasound clustering value, and the first input step is performed again to obtain the skin optical coherence tomography clustering image set.

[0057] Step 105: input the skin ultrasound cluster image set and the skin optical coherence tomography cluster image set into a multimodal image registration model to obtain a multimodal registered skin image set.

[0058] In some embodiments, the execution subject may input the skin ultrasound cluster image set and the skin optical coherence tomography cluster image set into a multimodal image registration model to obtain a multimodal registered skin image set. The multimodal image registration model may be a convolutional neural network that performs image registration on the input skin ultrasound cluster image and the corresponding skin optical coherence tomography cluster image and outputs the registered skin image. The multimodal registered skin image in the multimodal registered skin image set may be an image that combines the depth information of the skin ultrasound cluster image and the resolution of the skin optical coherence tomography cluster image.

[0059] In some optional implementations of some embodiments, the multimodal image registration model includes: a first downsampling attention mechanism network, a second downsampling attention mechanism network, a third downsampling attention mechanism network, a fourth downsampling attention mechanism network, a maximum pooling layer and a skin space conversion network. Among them, the first downsampling attention mechanism layer can be a network including two convolutional layers, a skin channel attention mechanism layer and a maximum pooling layer. The two convolutional layers can be convolutional layers with 32 convolution kernels of 3*3, a step size of 2, and an activation function of LeakyReLU (Leaky Rectified Linear Unit) activation function. The skin channel attention mechanism can be a module that obtains two 1*1*C feature maps by global maximum pooling and global average pooling of the feature map output by the input convolutional layer, and then inputs the two 1*1*C feature maps into a multilayer perceptron (MLP) with shared weights to sum the two 1*1*C feature maps element by element and output the channel weight coefficient, and finally multiplies the channel weight coefficient and the feature map output by the convolutional layer to obtain the output feature map. Wherein, C can be a predetermined threshold of the number of channels of the output feature map. The above-mentioned maximum pooling layer can be a maximum pooling layer with a convolution kernel of 2*2 and a stride of 2. The above-mentioned second downsampling attention mechanism network, the above-mentioned third downsampling attention mechanism network and the above-mentioned fourth downsampling attention mechanism network can be networks with the same network layers as the above-mentioned first downsampling attention mechanism network, but with different sizes of input feature maps. The above-mentioned skin space transformation network can be a convolutional network that performs geometric transformation (distortion transformation such as translation, scaling, rotation, etc.) on the feature map output by the input maximum pooling layer.

[0060] Optionally, the step of inputting the skin ultrasound cluster image set and the skin optical coherence tomography cluster image set into a multimodal image registration model to obtain a multimodal registered skin image set may include the following steps:

[0061] In the first step, each skin ultrasound cluster image in the skin ultrasound cluster image set and the corresponding skin optical coherence tomography cluster image in the skin optical coherence tomography cluster image set are channel-joined to generate a jointed skin image, thereby obtaining a jointed skin image set. The jointed skin image may be a skin image obtained by joining the channels of the skin ultrasound cluster image and the corresponding skin ultrasound cluster image.

[0062] In the second step, for each spliced ​​skin image in the spliced ​​skin image set, the following second input step is performed:

[0063] Sub-step 1, inputting the above-mentioned spliced ​​skin image into the above-mentioned first downsampling attention mechanism layer to obtain a first skin feature map group, wherein the above-mentioned first downsampling attention mechanism network includes: a downsampling convolution layer, a skin channel attention mechanism layer and a maximum pooling layer. The first skin feature map in the above-mentioned first skin feature map group can represent the information of the color, texture and depth features of the above-mentioned spliced ​​skin image.

[0064] Sub-step 2: Input the first skin feature map group into the second downsampling attention mechanism layer to obtain a second skin feature map group. The size of the second skin feature map in the second skin feature map group may be a feature map of half the size of the first skin feature map.

[0065] Sub-step 3: Input the second skin feature map group into the third downsampling attention mechanism layer to obtain a third skin feature map group. The size of the third skin feature map in the third skin feature map group may be a feature map of half the size of the second skin feature map.

[0066] Sub-step 4: Input the third skin feature map group into the fourth downsampling attention mechanism layer to obtain a fourth skin feature map group. The size of the fourth skin feature map in the fourth skin feature map group may be a feature map of half the size of the third skin feature map.

[0067] Sub-step 5: Input the fourth skin feature map group into the maximum pooling layer to obtain a fifth skin feature map group. The size of the fifth skin feature map in the fifth skin feature map group may be a feature map of half the size of the fourth skin feature map.

[0068] Sub-step 6, upsampling the fifth skin feature map group to obtain a sixth skin feature map group, wherein the size of the sixth skin feature map in the sixth skin feature map group may be the same as that of the first skin feature map.

[0069] Sub-step 7, inputting the sixth skin feature map group and the spliced ​​skin image into the skin space conversion network to obtain a multi-modal registered skin image.

[0070] In some optional implementations of some embodiments, the multimodal image registration model includes: a first upsampling calibration convolution layer, a second upsampling calibration convolution layer, a third upsampling calibration convolution layer, a fourth upsampling calibration convolution layer and a feature convolution layer. Among them, the first upsampling calibration convolution layer may include two calibration convolution layers. The calibration convolution layer may be a convolution layer with 32 convolution kernels of 3*3, a step size of 2, and an activation function of LeakyReLU (Leaky Rectified Linear Unit) activation function.

[0071] Optionally, performing upsampling processing on the fifth skin feature map group to obtain a sixth skin feature map group may include the following steps:

[0072] In the first step, the fifth skin feature map group is subjected to a deconvolution upsampling process to obtain a seventh skin feature map group. The size of the seventh skin feature map in the seventh skin feature map group may be a feature map with the same size as the fourth skin feature map. The deconvolution upsampling process may be performed by using a deconvolution network layer to perform a deconvolution process.

[0073] The second step is to perform feature splicing on the seventh skin feature map group and the fourth skin feature map group to obtain a first spliced ​​skin feature map group. The feature splicing may be feature splicing on channels.

[0074] The third step is to input the first spliced ​​skin feature map group into the first upsampling calibration convolution layer to obtain a second spliced ​​skin feature map group.

[0075] In the fourth step, the second spliced ​​skin feature map group is subjected to deconvolution upsampling processing to obtain a third spliced ​​skin feature map group, wherein the size of the third spliced ​​feature map in the third spliced ​​skin feature map group may be the same as that of the third skin feature map.

[0076] The fifth step is to perform feature splicing on the third spliced ​​skin feature map group and the third skin feature map group to obtain a fourth spliced ​​skin feature map group.

[0077] In the sixth step, the fourth spliced ​​skin feature map group is input into the second upsampling calibration convolution layer to obtain a fifth spliced ​​skin feature map group.

[0078] In the seventh step, the fifth spliced ​​skin feature map group is subjected to deconvolution upsampling processing to obtain a sixth spliced ​​skin feature map group, wherein the size of the sixth spliced ​​skin feature map in the sixth spliced ​​skin feature map group may be the same as that of the second skin feature map.

[0079] In the eighth step, feature stitching is performed on the sixth stitched skin feature map group and the second skin feature map group to obtain a seventh stitched skin feature map group.

[0080] In the ninth step, the seventh spliced ​​skin feature map group is input into the second upsampling calibration convolution layer to obtain an eighth spliced ​​skin feature map group.

[0081] In the tenth step, the eighth spliced ​​skin feature map group is subjected to deconvolution upsampling processing to obtain a ninth spliced ​​skin feature map group, wherein the size of the ninth spliced ​​skin feature map in the ninth spliced ​​skin feature map group may be the same as that of the first skin feature map.

[0082] In the eleventh step, feature stitching is performed on the ninth stitched skin feature map group and the first skin feature map group to obtain a tenth stitched skin feature map group.

[0083] In the twelfth step, the tenth spliced ​​skin feature map group is input into the fourth upsampling calibration convolution layer to obtain the eleventh spliced ​​skin feature map group.

[0084] In the thirteenth step, the eleventh spliced ​​skin feature map group is input into the feature convolution layer to obtain the sixth skin feature map group.

[0085] Step 106 , performing image block division processing on each multimodal registered skin image in the multimodal registered skin image set to obtain a multimodal registered skin image block set.

[0086] In some embodiments, the execution subject may perform image block division processing on each multimodal registered skin image in the multimodal registered skin image set to obtain a multimodal registered skin image block set. The multimodal registered skin image blocks in the multimodal registered skin image block set may be image blocks of a preset size that fuse the depth information of the skin ultrasound image and the skin optical coherence tomography image. The preset size may be a preset value. It should be noted that the preset size may be determined according to actual conditions. For example, the preset size may be 8*8.

[0087] Step 107 , performing Bayesian displacement compensation processing on the multimodal registered skin image set according to the multimodal registered skin image block set to obtain a multimodal motion compensated skin image set.

[0088] In some embodiments, the execution subject may perform Bayesian displacement compensation processing on the multimodal registered skin image set according to the multimodal registered skin image block set to obtain a multimodal motion compensated skin image set. The multimodal motion compensated skin image in the multimodal motion compensated skin image set may be an image that eliminates artifacts in the multimodal registered skin image and displacement differences caused by breathing and slight shaking of the target user.

[0089] As an example, the execution subject may utilize a motion tracking algorithm based on Bayesian reasoning to perform Bayesian displacement compensation processing on the multimodal registered skin image set according to the multimodal registered skin image block set to obtain a multimodal motion compensated skin image set. The motion tracking algorithm based on Bayesian reasoning may be to first determine the skin image block similarity values ​​between the multimodal registered skin image blocks of the multimodal registered skin image to obtain a skin image block similarity value set. Then, utilizing a three-step search algorithm, the multimodal registered skin image set is subjected to displacement compensation processing according to the skin image block similarity value set to obtain a multimodal motion compensated skin image set.

[0090] In the process of adopting technical solutions to solve the above-mentioned technical problem one, the following technical problem two is often accompanied: due to the breathing and slight shaking of the human body in skin imaging, the generated skin imaging has displacement and peak jump errors of the received signal, resulting in low quality of the generated skin image. For the above-mentioned technical problem two, the conventional solution is generally: using a region growing motion tracking algorithm to perform displacement compensation processing on the multimodal registered skin image set to obtain a displacement compensated skin image. However, the above-mentioned conventional solution still has the following technical problems: because the region growing motion tracking algorithm uses a similarity metric to determine the best matching position, it cannot solve the complex multimodal ultrasonic optical signals in the skin image and accurately predict the motion situation, resulting in low accuracy of displacement compensation and low quality of the multimodal registered skin image. The inventors have taken into account the shortcomings of conventional solutions and combined with the advantages / technical status of the image motion compensation algorithm owned by the inventors, and decided to adopt the following solution:

[0091] In some optional implementations of some embodiments, the above-mentioned multimodal registered skin image set is subjected to Bayesian displacement compensation processing according to the above-mentioned multimodal registered skin image block set to obtain a multimodal motion compensated skin image set, which may include the following steps:

[0092] In the first step, for each multimodal registered skin image patch in the multimodal registered skin image patch set, the following guided search steps are performed:

[0093] Sub-step 1: determining at least one multimodal registered skin image block that matches the multimodal registered skin image block as a target multimodal registered skin image block set. The determination may be performed by a block matching algorithm.

[0094] Sub-step 2, determining the normalized cross-correlation value of the multimodal registered skin image block and each target multimodal registered skin image block in the target multimodal registered skin image block set, and obtaining a normalized cross-correlation value set. The normalized cross-correlation value can characterize the similarity between the multimodal registered skin image block and the target multimodal registered skin image block. The determination can be performed by a normalized cross-correlation matching algorithm.

[0095] Sub-step 3, determining the adjacent displacement information between the multimodal registered skin image block and each adjacent multimodal registered skin image block in the adjacent multimodal registered skin image block set, and obtaining an adjacent displacement information set, wherein the adjacent multimodal registered skin image block may be an image block located above, below, left, and right of the multimodal registered skin image block. The adjacent displacement information in the adjacent displacement information set may represent the information of the motion vectors of the multimodal registered skin image block and the adjacent multimodal registered skin image block.

[0096] Sub-step 4, performing Bayesian estimation on the multimodal registered skin image block according to the normalized mutual correlation value set and the adjacent displacement information set, to obtain an image block matching displacement information set. The image block matching displacement information in the image block matching displacement information set can represent the information of the motion vector of the multimodal registered skin image block and the multimodal registered skin image blocks other than the multimodal registered skin image block.

[0097] As an example, the execution subject may use a Bayesian estimation algorithm to perform Bayesian estimation on the multimodal registered skin image block according to the normalized mutual correlation value set and the adjacent displacement information set to obtain an image block matching displacement information set.

[0098] Sub-step 5, according to the image block matching displacement information set, the image pixel point set included in the multimodal registered skin image block is subjected to regional displacement fitting processing to obtain a pixel displacement information set. The pixel displacement information in the pixel displacement information set may be the displacement information of each image pixel point determined according to the image block matching displacement information set of the image pixel point set.

[0099] As an example, the execution subject may use the least square method to perform regional displacement fitting processing on the image pixel point set included in the multimodal registered skin image block according to the image block matching displacement information set to obtain a pixel displacement information set.

[0100] Sub-step 6, based on the pixel displacement information set and the image block matching displacement information set, identify abnormal pixel values ​​of the image pixel point set to obtain an abnormal image pixel point set. The abnormal image pixel points in the abnormal image pixel point set may be image pixel points that deviate from the regularity of the image pixel points and displacement information corresponding to the image block matching displacement information set.

[0101] As an example, the execution subject may first determine the image block matching displacement information corresponding to the pixel displacement information set in the image block matching displacement information set as the target pixel displacement information set. Then, determine the position difference between the pixel displacement information set and the target pixel displacement information set to obtain a displacement difference set. Finally, at least one image pixel point in the image pixel point set whose corresponding displacement difference is not within a preset displacement difference threshold is taken as an abnormal image pixel point set. The preset displacement difference threshold may be a pre-set displacement difference range.

[0102] Sub-step 7: removing the abnormal image pixel point set from the image pixel point set to obtain a remaining image pixel point set.

[0103] Sub-step 8, screening the above-mentioned residual image pixel point set to obtain a target seed image pixel point set. Among them, the target seed image pixel points in the above-mentioned target seed image pixel point set can be the residual image pixel points whose posterior probability density corresponding to the residual image pixel points is greater than a preset posterior probability density threshold, and whose displacement difference is less than a preset displacement difference threshold. The above-mentioned preset posterior probability density threshold and preset displacement difference threshold can be pre-set values.

[0104] Sub-step 9, based on the target seed image pixel point set, the image pixel point set is subjected to guided search processing to obtain a target optimized displacement information set, wherein the target optimized displacement information in the target optimized displacement information set may be the optimal displacement information corresponding to the image pixel point.

[0105] As an example, the execution subject may utilize a region growing algorithm to perform a guided search process on the image pixel point set according to the target seed image pixel point set to obtain a target optimized displacement information set.

[0106] In the second step, according to the obtained displacement information sets of each target optimization, Bayesian displacement compensation processing is performed on the above multimodal registered skin image set to obtain a multimodal motion compensated skin image set.

[0107] The above technical solution and its related contents, as an inventive point of an embodiment of the present disclosure, solve the second technical problem mentioned in the background technology: "Since the region growing motion tracking algorithm uses a similarity metric to determine the best matching position, it cannot solve the complex multimodal ultrasonic optical signals in the skin image and accurately predict the motion situation, resulting in low accuracy of displacement compensation and low quality of multimodal registered skin images." The factors that lead to low accuracy of displacement compensation and low quality of multimodal registered skin images are often as follows: Since the region growing motion tracking algorithm uses a similarity metric to determine the best matching position, it cannot solve the complex multimodal ultrasonic optical signals in the skin image and accurately predict the motion situation. If the above factors are solved, the effect of improving the accuracy of displacement compensation and the quality of multimodal registered skin images can be achieved. In order to achieve this effect, the present disclosure first optimizes the displacement by simultaneously using the normalized mutual information and the Bayesian optimization algorithm of the motion of adjacent multimodal registered skin image blocks, and by converting the traditional similarity metric into a probability distribution and combining the information of adjacent multimodal registered skin image blocks, the error caused by the displacement determined by a single matching block is avoided, so that the displacement estimation is more accurate and reliable, and the influence of abnormal situations such as peak jump error on the displacement estimation is reduced. Then, the multimodal registered skin image block is subjected to Bayesian estimation for regional displacement fitting processing to identify abnormal image pixel points. Based on the assumption of tissue motion continuity, the abnormal displacement points that do not conform to the displacement distribution law of the surrounding points are detected and corrected by local displacement fitting through two-dimensional polynomial fitting, which can reduce the noise and error in the multimodal registered skin image and improve the image smoothness. Finally, by selecting the target seed image pixel point set and performing the regional growth algorithm for guided search of the displacement of the target seed image pixel point, the local accurate displacement information can be fully utilized, the regional range of the accurately estimated displacement can be gradually expanded, and the local displacement estimation results can be integrated into the complete displacement information of the entire region of interest to achieve motion tracking of the entire region, which can reduce a large number of calculation steps, simplify the calculation complexity of displacement compensation, improve the displacement compensation efficiency and displacement accuracy of the multimodal registered skin image set, and then improve the quality of the multimodal motion compensated skin image set, and reduce the artifacts in the multimodal motion compensated skin image set and the image blur caused by slight shaking of the human body.

[0108] Step 108: performing terminal image display on the multimodal motion compensated skin image set.

[0109] In some embodiments, the execution subject may perform terminal image display on the multimodal motion compensated skin image set. The terminal highlight display may be sending the multimodal motion compensated image set to a target terminal for image display. The target terminal may be a display screen terminal.

[0110] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the multimodal motion-compensated skin image display method of some embodiments of the present disclosure can improve the quality of the skin image and reduce the artifacts in the image caused by the human body's micro-breathing through the registration and motion compensation of the multimodal images. Specifically, the quality of the related motion-compensated image is lower, and the motion-compensated image may still have some artifacts and blurring problems that cannot be removed. More imaging images are required under the condition of ensuring the quality of skin features, which increases the waste of transmission resources. The reason is that: motion compensation is performed by a single type of image, and the single type of image itself contains less skin feature information, resulting in lower quality of the motion-compensated image, and the motion-compensated image may still have some artifacts and blurring problems that cannot be removed. More imaging images are required under the condition of ensuring the quality of skin features, which increases the waste of transmission resources. Based on this, the multimodal motion-compensated skin image display method of some embodiments of the present disclosure can first perform probe space calibration processing on the ultrasonic probe and the optical coherence tomography probe in the skin elasticity detection system to obtain a calibrated ultrasonic probe and a calibrated optical coherence tomography probe. Here, the coordinate systems corresponding to the ultrasonic probe and the optical coherence tomography probe are converted to the coordinate system of the target user, the positioning accuracy of the probe is improved, and the position information and direction information of the probe are integrated into the imaging image, thereby improving the accuracy of subsequent imaging positioning scanning. Secondly, the skin scanning path information of the calibrated ultrasonic probe and the calibrated optical coherence tomography probe in the target user's skin to be detected area is determined. Here, a more accurate probe scanning path can be determined, which is convenient for subsequent control of the scanning of the optical coherence tomography probe and the ultrasonic probe, and reduces the overlap between the generated images. Thirdly, according to the skin scanning path information, the calibrated ultrasonic probe and the calibrated optical coherence tomography probe are controlled to perform probe scanning imaging processing in the skin to be detected area to obtain a skin ultrasonic image set and a skin optical coherence tomography image set. Here, the overlap of skin feature information between the skin ultrasonic image in the generated skin ultrasonic image set and the skin optical coherence tomography image in the skin optical coherence tomography image set can be reduced, and redundant information and data volume can be reduced, so as to improve the efficiency of subsequent registration and motion compensation. Subsequently, gamma hybrid clustering processing is performed on each skin ultrasound image in the skin ultrasound image set and each skin optical coherence tomography image in the skin optical coherence tomography image set to obtain a skin ultrasound cluster image set and a skin optical coherence tomography cluster image set. Here, the detailed information of different skin tissue structures in the skin ultrasound image and the skin optical coherence tomography image can be accurately identified, and the determination of the position correspondence relationship of the subsequent image registration is facilitated by considering the probability distribution of the gray value. Then, the skin ultrasound cluster image set and the skin optical coherence tomography cluster image set are input into the multimodal image registration model to obtain a multimodal registered skin image set.Here, the characteristic information of skin ultrasound clustering images and skin optical coherence tomography clustering images can be automatically learned through the multimodal image registration model to improve the accuracy of multimodal image registration. Afterwards, each multimodal registered skin image in the above multimodal registered skin image set is subjected to image block division processing to obtain a multimodal registered skin image block set. Here, the multimodal registered skin image block set is convenient for subsequent motion compensation based on image blocks to improve the quality of the image. Then, according to the above multimodal registered skin image block set, the above multimodal registered skin image set is subjected to Bayesian displacement compensation processing to obtain a multimodal motion compensated skin image set. Here, the artifact problem in the multimodal registered image can be effectively removed, the influence of noise on the image can be reduced, and the quality of the multimodal motion compensated skin image can be improved. Finally, the terminal image display of the above multimodal motion compensated skin image set is performed. Here, since the multimodal motion compensated skin image contains more skin detail information and tissue structure information, the number of skin images can be reduced under the same skin feature information, thereby reducing the waste of transmission resources. It can be concluded that the multimodal motion compensated skin image display method can improve the quality of skin images and reduce artifacts in the image caused by human body's micro-breathing through multimodal image registration and motion compensation.

[0111] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a multi-modal motion compensation skin image display device, and these device embodiments are Figure 1 Corresponding to the method embodiments shown, the multimodal motion-compensated skin image display device can be specifically applied to various electronic devices.

[0112] like Figure 2As shown, a multimodal motion compensated skin image display device 200 includes: a probe space calibration unit 201, a determination unit 202, a control unit 203, a gamma hybrid clustering unit 204, an input unit 205, an image block division unit 206, a Bayesian displacement compensation unit 207 and an image display unit 208. Among them, the probe space calibration unit 201 is configured to: perform probe space calibration processing on the ultrasound probe and the optical coherence tomography probe in the skin elasticity detection system to obtain the calibrated ultrasound probe and the calibrated optical coherence tomography probe. The determination unit 202 is configured to: determine the skin scanning path information of the above-mentioned calibrated ultrasound probe and the above-mentioned calibrated optical coherence tomography probe in the skin to be detected area of ​​the target user. The control unit 203 is configured to: according to the above-mentioned skin scanning path information, control the above-mentioned calibrated ultrasound probe and the above-mentioned calibrated optical coherence tomography probe to perform probe scanning imaging processing in the above-mentioned skin to be detected area to obtain a skin ultrasound image set and a skin optical coherence tomography image set. The gamma hybrid clustering unit 204 is configured to perform gamma hybrid clustering processing on each skin ultrasound image in the above-mentioned skin ultrasound image set and each skin optical coherence tomography image in the above-mentioned skin optical coherence tomography image set to obtain a skin ultrasound clustering image set and a skin optical coherence tomography clustering image set. The input unit 205 is configured to input the above-mentioned skin ultrasound clustering image set and the above-mentioned skin optical coherence tomography clustering image set into a multimodal image registration model to obtain a multimodal registered skin image set. The image block division unit 206 is configured to perform image block division processing on each multimodal registered skin image in the above-mentioned multimodal registered skin image set to obtain a multimodal registered skin image block group set. The Bayesian displacement compensation unit 207 is configured to perform Bayesian displacement compensation processing on the above-mentioned multimodal registered skin image set according to the above-mentioned multimodal registered skin image block group set to obtain a multimodal motion compensated skin image set. The image display unit 208 is configured to perform terminal image display on the multimodal motion compensated skin image set.

[0113] It can be understood that the units described in the multimodal motion compensation skin image display device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the multimodal motion compensated skin image display device 200 and the units included therein, and will not be described in detail here.

[0114] Reference below Figure 3 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0115] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0116] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0117] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0118] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0119] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0120] The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist independently without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: performs probe space calibration processing on the ultrasonic probe and the optical coherence tomography probe in the skin elasticity detection system to obtain the calibrated ultrasonic probe and the calibrated optical coherence tomography probe; determines the skin scanning path information of the above-mentioned calibrated ultrasonic probe and the above-mentioned calibrated optical coherence tomography probe in the target user's skin to-be-detected area; according to the above-mentioned skin scanning path information, controls the above-mentioned calibrated ultrasonic probe and the above-mentioned calibrated optical coherence tomography probe to perform probe scanning imaging processing in the above-mentioned skin to-be-detected area to obtain a skin ultrasonic image set and a skin optical coherence tomography image set; for each skin ultrasonic image set in the above-mentioned skin ultrasonic image set, The ultrasonic image and each skin optical coherence tomography image in the above-mentioned skin optical coherence tomography image set are subjected to gamma hybrid clustering processing to obtain a skin ultrasound clustering image set and a skin optical coherence tomography clustering image set; the above-mentioned skin ultrasound clustering image set and the above-mentioned skin optical coherence tomography clustering image set are input into a multimodal image registration model to obtain a multimodal registered skin image set; each multimodal registered skin image in the above-mentioned multimodal registered skin image set is subjected to image block division processing to obtain a multimodal registered skin image block group set; according to the above-mentioned multimodal registered skin image block group set, the above-mentioned multimodal registered skin image set is subjected to Bayesian displacement compensation processing to obtain a multimodal motion compensated skin image set; the above-mentioned multimodal motion compensated skin image set is subjected to terminal image display.

[0121] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0123] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be arranged in a processor, for example, may be described as: a processor including a probe space calibration unit, a determination unit, a control unit, a gamma hybrid clustering unit, an input unit, an image block division unit, a Bayesian displacement compensation unit and an image display unit. Among them, the names of these units do not constitute a limitation on the units themselves under certain circumstances. For example, the probe space calibration unit may also be described as a "unit that performs probe space calibration processing on the ultrasound probe and the optical coherence tomography probe in the skin elasticity detection system to obtain a calibrated ultrasound probe and a calibrated optical coherence tomography probe".

[0124] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0125] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A multimodal motion compensated skin image display method, comprising: Performing probe space calibration processing on an ultrasound probe and an optical coherence tomography probe in a skin elasticity detection system to obtain a calibrated ultrasound probe and a calibrated optical coherence tomography probe; Determining skin scanning path information of the calibrated ultrasound probe and the calibrated optical coherence tomography probe in a skin area to be detected of a target user; According to the skin scanning path information, controlling the calibrated ultrasound probe and the calibrated optical coherence tomography probe to perform probe scanning imaging processing on the skin to-be-detected area to obtain a skin ultrasound image set and a skin optical coherence tomography image set; Performing gamma hybrid clustering processing on each skin ultrasound image in the skin ultrasound image set and each skin optical coherence tomography image in the skin optical coherence tomography image set to obtain a skin ultrasound clustering image set and a skin optical coherence tomography clustering image set; Inputting the skin ultrasound cluster image set and the skin optical coherence tomography cluster image set into a multimodal image registration model to obtain a multimodal registered skin image set; Performing image block division processing on each multimodal registered skin image in the multimodal registered skin image set to obtain a multimodal registered skin image block set; According to the multimodal registered skin image block set, performing Bayesian displacement compensation processing on the multimodal registered skin image set to obtain a multimodal motion compensated skin image set; The multimodal motion compensated skin image set is subjected to terminal image display.

2. The method according to claim 1, wherein: The method of performing probe space calibration processing on the ultrasound probe and the optical coherence tomography probe in the skin elasticity detection system to obtain a calibrated ultrasound probe and a calibrated optical coherence tomography probe comprises: Acquire an ultrasound calibration skin image corresponding to the ultrasound probe and an optical coherence tomography calibration skin image corresponding to the optical coherence tomography probe; Extracting corner points from the ultrasonically calibrated skin image and the optical coherence tomography-calibrated skin image to obtain an ultrasonic skin corner point information set and an optical skin corner point information set; Determine the three-dimensional ultrasonic coordinate information of each ultrasonic skin corner point information in the skin to-be-detected area in the ultrasonic skin corner point information set to obtain a three-dimensional ultrasonic coordinate information set; Determine the three-dimensional optical coordinate information of each optical skin corner point information in the optical skin corner point information set in the skin to-be-detected area to obtain a three-dimensional optical coordinate information set; Performing spatial transformation on each ultrasonic skin corner point information in the ultrasonic skin corner point information set and the three-dimensional ultrasonic coordinate information corresponding to the three-dimensional ultrasonic coordinate information set to obtain an ultrasonic space transformation matrix; Performing spatial transformation on each optical skin corner point information in the optical skin corner point information set and the three-dimensional space ultrasound coordinate information corresponding to the three-dimensional space optical coordinate information set to obtain an optical space transformation matrix; According to the ultrasonic space conversion matrix and the optical space conversion matrix, the ultrasonic probe and the optical coherence tomography probe are spatially calibrated to obtain a calibrated ultrasonic probe and a calibrated optical coherence tomography probe.

3. The method according to claim 1, wherein: The step of performing gamma hybrid clustering processing on each skin ultrasound image in the skin ultrasound image set and each skin optical coherence tomography image in the skin optical coherence tomography image set to obtain a skin ultrasound cluster image set and a skin optical coherence tomography cluster image set comprises: Based on the skin ultrasound image set, the following first input step is performed: determining an ultrasound clustering value for the skin ultrasound image set and an optical clustering value for the skin optical coherence tomography image set; Performing image clustering processing on the skin ultrasound image set according to the ultrasound clustering value to obtain a skin ultrasound image cluster set; Based on each skin ultrasound image cluster in the skin ultrasound image cluster set, the following optimization steps are performed: Determine a ratio of the number of skin ultrasound images included in the skin ultrasound image cluster to the number of skin ultrasound images as an initial gamma mixing weight value, wherein the number of skin ultrasound images is the number of skin ultrasound images included in the skin ultrasound image set; Determine a pixel mean of each skin ultrasound image included in the skin ultrasound image cluster as an initial gamma mixed mean; Determine the square root of the mean of the sum of squares of differences between the pixel values ​​of each skin ultrasound image and the pixel mean as an initial gamma mixing standard deviation; Determine the initial gamma mixing weight value, the initial gamma mixing mean value and the initial gamma mixing standard deviation as a mixing model parameter set of an initial gamma mixing model for the skin ultrasound image cluster; Optimizing the initial gamma mixed model according to the mixed model parameter set using a maximum expectation algorithm to obtain an optimized gamma mixed model; Inputting the skin ultrasound image cluster set into the optimized gamma mixture model to obtain a skin ultrasound cluster image set; The skin optical coherence tomography image set and the optical clustering value are determined as the skin ultrasound image set and the ultrasound clustering value, and the first input step is performed again to obtain a skin optical coherence tomography clustering image set.

4. The method according to claim 1, wherein: The multimodal image registration model includes: a first downsampling attention mechanism network, a second downsampling attention mechanism network, a third downsampling attention mechanism network, a fourth downsampling attention mechanism network, a maximum pooling layer and a skin space transformation network; and The step of inputting the skin ultrasound cluster image set and the skin optical coherence tomography cluster image set into a multimodal image registration model to obtain a multimodal registered skin image set comprises: Performing channel stitching on each skin ultrasound cluster image in the skin ultrasound cluster image set and the corresponding skin optical coherence tomography cluster image in the skin optical coherence tomography cluster image set to generate a stitched skin image, thereby obtaining a stitched skin image set; For each stitched skin image in the stitched skin image set, the following second input step is performed: Inputting the spliced ​​skin image into the first downsampling attention mechanism network to obtain a first skin feature map group, wherein the first downsampling attention mechanism network includes: a downsampling convolution layer, a skin channel attention mechanism layer and a maximum pooling layer; Inputting the first skin feature map group into the second downsampling attention mechanism network to obtain a second skin feature map group; Inputting the second skin feature map group into the third downsampling attention mechanism network to obtain a third skin feature map group; Inputting the third skin feature map group into the fourth downsampling attention mechanism network to obtain a fourth skin feature map group; Inputting the fourth skin feature map group into the maximum pooling layer to obtain a fifth skin feature map group; Performing upsampling processing on the fifth skin feature map group to obtain a sixth skin feature map group; The sixth skin feature map group and the spliced ​​skin image are input into the skin space conversion network to obtain a multimodal registered skin image.

5. The method according to claim 4, wherein: The multimodal image registration model comprises: a first upsampling calibration convolution layer, a second upsampling calibration convolution layer, a third upsampling calibration convolution layer, a fourth upsampling calibration convolution layer and a feature convolution layer; and The upsampling process is performed on the fifth skin feature map group to obtain a sixth skin feature map group, comprising: Performing a deconvolution upsampling process on the fifth skin feature map group to obtain a seventh skin feature map group; Performing feature splicing on the seventh skin feature map group and the fourth skin feature map group to obtain a first spliced ​​skin feature map group; Inputting the first spliced ​​skin feature map group into the first upsampling calibration convolution layer to obtain a second spliced ​​skin feature map group; Performing a deconvolution upsampling process on the second spliced ​​skin feature map group to obtain a third spliced ​​skin feature map group; Performing feature stitching on the third stitched skin feature map group and the third skin feature map group to obtain a fourth stitched skin feature map group; Inputting the fourth spliced ​​skin feature map group into the second upsampling calibration convolution layer to obtain a fifth spliced ​​skin feature map group; Performing a deconvolution upsampling process on the fifth spliced ​​skin feature map group to obtain a sixth spliced ​​skin feature map group; Performing feature splicing on the sixth spliced ​​skin feature map group and the second skin feature map group to obtain a seventh spliced ​​skin feature map group; Inputting the seventh spliced ​​skin feature map group into the second upsampling calibration convolution layer to obtain an eighth spliced ​​skin feature map group; Performing a deconvolution upsampling process on the eighth spliced ​​skin feature map group to obtain a ninth spliced ​​skin feature map group; Performing feature splicing on the ninth spliced ​​skin feature map group and the first skin feature map group to obtain a tenth spliced ​​skin feature map group; Inputting the tenth spliced ​​skin feature map group into the fourth upsampling calibration convolution layer to obtain an eleventh spliced ​​skin feature map group; The eleventh spliced ​​skin feature map group is input into the feature convolution layer to obtain a sixth skin feature map group.

6. A multimodal motion-compensated skin image display device, comprising: A probe space calibration unit is configured to perform probe space calibration processing on an ultrasound probe and an optical coherence tomography probe in a skin elasticity detection system to obtain a calibrated ultrasound probe and a calibrated optical coherence tomography probe; a determination unit configured to determine skin scanning path information of the calibrated ultrasound probe and the calibrated optical coherence tomography probe in a skin area to be detected of a target user; A control unit is configured to control the calibrated ultrasound probe and the calibrated optical coherence tomography probe to perform probe scanning imaging processing on the skin area to be detected according to the skin scanning path information, so as to obtain a skin ultrasound image set and a skin optical coherence tomography image set; a gamma hybrid clustering unit configured to perform gamma hybrid clustering processing on each skin ultrasound image in the skin ultrasound image set and each skin optical coherence tomography image in the skin optical coherence tomography image set to obtain a skin ultrasound clustering image set and a skin optical coherence tomography clustering image set; An input unit, configured to input the skin ultrasound cluster image set and the skin optical coherence tomography cluster image set into a multimodal image registration model to obtain a multimodal registered skin image set; An image block division unit is configured to perform image block division processing on each multimodal registered skin image in the multimodal registered skin image set to obtain a multimodal registered skin image block set; A Bayesian displacement compensation unit is configured to perform Bayesian displacement compensation processing on the multimodal registered skin image set according to the multimodal registered skin image block set to obtain a multimodal motion compensated skin image set; The image display unit is configured to perform terminal image display on the multimodal motion compensated skin image set.

7. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images