Calibration Method and Device, Electronic Device, and Computer-Readable Storage Medium

Abstract

The present application discloses a calibration method and apparatus, an electronic device, and a computer-readable storage medium. The method includes: obtaining a first trajectory of an imaging device and a target image sequence, where the target image sequence is an image sequence obtained by the imaging device capturing a reference object during movement along the first trajectory; obtaining a second trajectory of the reference object in the target image sequence according to the position of the reference object in the target image sequence; and obtaining the imaging delay time of the imaging device by aligning the first trajectory with the second trajectory.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to a calibration method and apparatus, an electronic device, and a computer-readable storage medium. Background Art

[0002] Currently, imaging devices generally have the characteristic of imaging delay. Specifically, there is a time difference between when the imaging device starts to collect an image and when the image is generated. That is to say, the timestamp of the image collected by the imaging device is inconsistent with the time when the imaging device starts to collect the image. Based on the imaging delay time of the imaging device, the timestamp of the image can be corrected to the time when the imaging device starts to collect the image. Therefore, how to determine the imaging delay time of the imaging device is of great significance. Summary of the Invention

[0003] This application provides a calibration method and apparatus, an electronic device, and a computer-readable storage medium.

[0004] In a first aspect, a calibration method is provided. The method includes:

[0005] Obtain a first trajectory of an imaging device and a target image sequence, where the target image sequence is a sequence of images obtained by the imaging device capturing a reference object during movement along the first trajectory;

[0006] Obtain a second trajectory of the reference object in the target image sequence according to the position of the reference object in the target image sequence;

[0007] Obtain the imaging delay time of the imaging device by aligning the first trajectory with the second trajectory.

[0008] In combination with any embodiment of this application, the obtaining the imaging delay time of the imaging device by aligning the first trajectory with the second trajectory includes:

[0009] When the first trajectory is aligned with the second trajectory by moving a target time along the time axis, determine that the imaging delay time of the imaging device is the target time.

[0010] In combination with any embodiment of this application, the obtaining the imaging delay time of the imaging device by aligning the first trajectory with the second trajectory includes:

[0011] Obtain a candidate delay time;

[0012] Add the candidate delay time to the timestamps of the trajectory points of the first trajectory to obtain a third trajectory;

[0013] When the second trajectory is aligned with the third trajectory, determine the imaging delay time of the imaging device as the candidate delay time.

[0014] In combination with any embodiment of the present application, before determining that the imaging delay time of the imaging device is the candidate delay time when the second trajectory is aligned with the third trajectory, the method further includes:

[0015] When the difference between the second trajectory and the third trajectory is less than or equal to a first threshold, determine that the second trajectory is aligned with the third trajectory.

[0016] In combination with any embodiment of the present application, the second trajectory includes a first inflection point, the trajectory points other than the first inflection point in the second trajectory are first non-inflection points, the third trajectory includes a second inflection point, and the trajectory points other than the second inflection point in the third trajectory are second non-inflection points;

[0017] Before determining that the imaging delay time of the imaging device is the candidate delay time when the second trajectory is aligned with the third trajectory, the method further includes:

[0018] When the difference between the first inflection point and the second inflection point is less than or equal to a second threshold and the difference between the first non-inflection point and the second non-inflection point is less than or equal to a third threshold, determine that the second trajectory is aligned with the third trajectory, where the second threshold is less than the third threshold.

[0019] In combination with any embodiment of the present application, before obtaining the second trajectory of the reference object in the image sequence according to the position of the reference object in the target image sequence, the method further includes:

[0020] Perform segmentation processing on the images in the target image sequence to obtain a segmentation result sequence of the target image sequence;

[0021] Determine the position of the reference object in the target image sequence according to the segmentation result sequence.

[0022] In combination with any embodiment of the present application, obtaining the second trajectory of the reference object in the image sequence according to the position of the reference object in the target image sequence includes:

[0023] Obtain a position sequence of the reference object according to the position of the reference object in the target image sequence;

[0024] Perform curve fitting on the position sequence of the reference object to obtain the second trajectory.

[0025] In combination with any embodiment of the present application, after obtaining the imaging delay time of the imaging device, the method further includes:

[0026] Obtaining a to-be-processed image and a target conversion relationship collected by the imaging device, where the to-be-processed image includes a target object, the time stamp of the to-be-processed image is a first time, and the target conversion relationship is a conversion relationship between the pixel coordinate system and the world coordinate system of the to-be-processed image;

[0027] According to the position of the target object in the to-be-processed image and the target conversion relationship, obtaining the target position of the target object in the world coordinate system as the position of the target object body at a second time, where the second time is the difference between the first time and the imaging delay time.

[0028] In a second aspect, a calibration device is provided, and the calibration device includes:

[0029] An acquisition unit, configured to acquire a first trajectory of an imaging device and a target image sequence, where the target image sequence is an image sequence obtained by the imaging device taking pictures of a reference object during the process of moving according to the first trajectory;

[0030] A processing unit, configured to obtain a second trajectory of the reference object in the target image sequence according to the position of the reference object in the target image sequence;

[0031] An alignment unit, configured to obtain the imaging delay time of the imaging device by aligning the first trajectory with the second trajectory.

[0032] In combination with any embodiment of the present application, the alignment unit is specifically configured to:

[0033] When aligning the first trajectory with the second trajectory by moving the target time along the time axis, determining that the imaging delay time of the imaging device is the target time.

[0034] In combination with any embodiment of the present application, the alignment unit is specifically configured to:

[0035] Obtain a candidate delay time;

[0036] Adding the candidate delay time to the time stamps of the trajectory points of the first trajectory to obtain a third trajectory;

[0037] When the second trajectory is aligned with the third trajectory, determining that the imaging delay time of the imaging device is the candidate delay time.

[0038] In combination with any embodiment of the present application, the calibration device further includes:

[0039] A determination unit, configured to determine that the second trajectory is aligned with the third trajectory when the difference between the second trajectory and the third trajectory is less than or equal to a first threshold.

[0040] Combined with any implementation manner of the present application, the second trajectory includes a first inflection point, the trajectory points other than the first inflection point in the second trajectory are first non-inflection points, the third trajectory includes a second inflection point, and the trajectory points other than the second inflection point in the third trajectory are second non-inflection points;

[0041] The determination unit is further configured to determine that the second trajectory is aligned with the third trajectory when the difference between the first inflection point and the second inflection point is less than or equal to a second threshold, and the difference between the first non-inflection point and the second non-inflection point is less than or equal to a third threshold, where the second threshold is less than the third threshold.

[0042] Combined with any implementation manner of the present application, the processing unit is further configured to:

[0043] Perform segmentation processing on the images in the target image sequence to obtain a segmentation result sequence of the target image sequence;

[0044] Determine the position of the reference object in the target image sequence according to the segmentation result sequence.

[0045] Combined with any implementation manner of the present application, the processing unit is specifically configured to:

[0046] Obtain a position sequence of the reference object according to the position of the reference object in the target image sequence;

[0047] Perform curve fitting on the position sequence of the reference object to obtain the second trajectory.

[0048] Combined with any implementation manner of the present application, the acquisition unit is further configured to acquire a to-be-processed image and a target conversion relationship collected by the imaging device, where the to-be-processed image includes a target object, the time stamp of the to-be-processed image is a first time, and the target conversion relationship is a conversion relationship between the pixel coordinate system and the world coordinate system of the to-be-processed image;

[0049] The processing unit is further configured to obtain a target position of the target object in the world coordinate system according to the position of the target object in the to-be-processed image and the target conversion relationship, and use it as the position of the target object body at a second time, where the second time is the difference between the first time and the imaging delay time.

[0050] In a third aspect, an electronic device is provided, which includes: a processor and a memory. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method according to the first aspect and any possible implementation manner thereof as described above.

[0051] In a fourth aspect, another electronic device is provided, which includes: a processor, a sending device, an input device, an output device, and a memory. The memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method according to the first aspect and any possible implementation manner thereof as described above.

[0052] In a fifth aspect, a computer-readable storage medium is provided. A computer program is stored in the computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to the first aspect and any possible implementation manner thereof as described above.

[0053] In a sixth aspect, a computer program product is provided. The computer program product includes a computer program or instructions. When the computer program or instructions run on a computer, the computer is caused to execute the method according to the first aspect and any possible implementation manner thereof.

[0054] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit this application.

[0055] In the embodiments of this application, a calibration device obtains a first trajectory of an imaging device and a target image sequence, where the target image sequence is an image sequence obtained by the imaging device taking pictures of a reference object during the process of moving along the first trajectory. Then, according to the position of the reference object in the target image sequence, a second trajectory of the reference object in the target image sequence can be obtained. Since there is relative movement between the imaging device and the reference object during the process of the imaging device collecting the target image sequence, by aligning the first trajectory with the second trajectory, the imaging delay time of the imaging device can be obtained. Description of the Drawings

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the following will describe the drawings required to be used in the embodiments of this application or the background art.

[0057] The drawings here are incorporated into the description and form a part of this description. These drawings show embodiments that conform to this application and, together with the description, are used to illustrate the technical solutions of this application.

[0058] Figure 1 Schematic flowchart of a calibration method provided by an embodiment of the present application;

[0059] Figure 2 Schematic diagram of a first trajectory and a second trajectory provided by an embodiment of the present application;

[0060] Figure 3 Another schematic diagram of a first trajectory and a second trajectory provided by an embodiment of the present application;

[0061] Figure 4 Schematic diagram of a strong echo line at the bottom of a water tank provided by an embodiment of the present application;

[0062] Figure 5 Schematic diagram of a phantom nylon line provided by an embodiment of the present application;

[0063] Figure 6 Schematic structural diagram of a calibration device provided by an embodiment of the present application;

[0064] Figure 7 Schematic hardware structure diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0065] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0066] The terms "first", "second", etc. in the specification and claims of the present application and the above accompanying drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0067] Reference to "embodiment" in this application means that the specific features, structures, or characteristics described in conjunction with the embodiment may be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment every time, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0068] The execution subject of the embodiment of this application is a calibration device. Among them, the calibration device can be any electronic device that can execute the technical solution disclosed in the method embodiment of this application. Optionally, the calibration device can be one of the following: a computer, a server.

[0069] It should be understood that the method embodiment of this application can also be implemented by a processor executing computer program code. The embodiments of this application will be described below in conjunction with the accompanying drawings in the embodiments of this application. Please refer to Figure 1 , Figure 1 is a schematic flowchart of a calibration method provided by the embodiment of this application.

[0070] 101. Obtain the first trajectory of the imaging device and the target image sequence.

[0071] In the embodiment of this application, the target image sequence is an image sequence obtained by the imaging device taking pictures of the reference object during the movement along the first trajectory. For example, the imaging device moves along the first trajectory and takes pictures of a reference object in a stationary state to obtain images a, b, and c. Among them, image a is an image obtained by the imaging device taking a picture of the reference object at the first trajectory point in the first trajectory, image b is an image obtained by the imaging device taking a picture of the reference object at the second trajectory point in the first trajectory, and image c is an image obtained by the imaging device taking a picture of the reference object at the third trajectory point in the first trajectory. The first trajectory point, the second trajectory point, and the third trajectory point are three different trajectory points, and images a, b, and c all include the reference object. The reference object can be any object. For example, the reference object is an imitation nylon thread, or the reference object is a strong echo line at the bottom of the water tank.

[0072] It should be understood that the images in the target image sequence all include time stamps, and the time stamps represent the time when the imaging device generates the images. For example, the target image sequence includes images a and b. Among them, the time stamp of image a is t1, and the time stamp of image b is t2. Then image a is generated by the imaging device at time t1, and image b is generated by the imaging device at time t2.

[0073] The imaging device can be any imageable device. Optionally, the imaging device is an ultrasonic probe, that is, the ultrasonic probe takes pictures of the reference object during the movement along the first trajectory to obtain the target image sequence.

[0074] In an implementation of obtaining an image to be processed, there is a communication connection between the calibration device and the imaging device, and the calibration device obtains a target image sequence collected by the imaging device through this communication connection.

[0075] In another implementation of obtaining a target image sequence, the calibration device receives the target image sequence input by the user through an input component, where the input component includes: a mouse, a keyboard, a touch screen, a touch pad, and an audio input device.

[0076] In yet another implementation of obtaining a target image sequence, the calibration device receives the target image sequence sent by a terminal, where the terminal includes: a mobile phone, a computer, a tablet computer, and a smart wearable device.

[0077] 102. According to the position of the above-mentioned reference object in the above-mentioned target image sequence, obtain the second trajectory of the above-mentioned reference object in the above-mentioned target image sequence.

[0078] In the embodiment of the present application, the second trajectory is the trajectory of the reference object in the target image sequence. For example, the target image sequence includes image a, image b, and image c, where the time stamp of image a is t1, the time stamp of image b is t2, and the time stamp of image c is t3. t1 is less than t2, and t2 is less than t3, that is, the generation time of image a is earlier than the generation time of image b, and the generation time of image b is earlier than the generation time of image c. If the position of the reference object in image a is p1, the position of the reference object in image b is p2, and the position of the reference object in image c is p3, then the trajectory of the reference object in the target image sequence is: the position at time t1 is p1, the position at time t2 is p2, and the position at time t3 is p3.

[0079] Optionally, the calibration device determines the position of the reference object in the target image sequence by performing target detection processing on the images in the target image sequence.

[0080] 103. By aligning the above-mentioned first trajectory with the above-mentioned second trajectory, obtain the imaging delay time of the above-mentioned imaging device.

[0081] During the process of the imaging device acquiring the target image sequence, the reference object is stationary and the imaging device is in motion. That is, during the process of the imaging device acquiring the target image sequence, there is relative motion between the reference object and the imaging device. Therefore, if the imaging device has no imaging delay time, the first trajectory should be the same as the second trajectory. That is to say, in the case where the imaging device has an imaging delay time, there is a time difference between the first trajectory and the second trajectory. Thus, the calibration device can determine the imaging delay time of the imaging device by aligning the first trajectory with the second trajectory. Specifically, the calibration device can determine the time difference between the first trajectory and the second trajectory by aligning the first trajectory with the second trajectory, and this time difference is the imaging delay time of the imaging device.

[0082] In the embodiment of the present application, the calibration device acquires the first trajectory of the imaging device and the target image sequence. Among them, the target image sequence is an image sequence obtained by the imaging device photographing the reference object during the process of moving along the first trajectory. Then, according to the position of the reference object in the target image sequence, the second trajectory of the reference object in the target image sequence can be obtained. Since there is relative motion between the imaging device and the reference object during the process of the imaging device acquiring the target image sequence, the imaging delay time of the imaging device can be obtained by aligning the first trajectory with the second trajectory.

[0083] As an optional implementation manner, the calibration device performs the following steps during the execution of step 103:

[0084] 201. When aligning the second trajectory by moving the first trajectory along the time axis by a target time, determine that the imaging delay time of the imaging device is the target time.

[0085] The calibration device can align the first trajectory with the second trajectory by moving the first trajectory along the time axis. When the first trajectory is aligned with the second trajectory, the target time of the movement of the first trajectory along the time axis is used as the delay time of the imaging device.

[0086] In a possible implementation manner, the calibration device displays the first trajectory and the second trajectory in the target coordinate system, where the horizontal axis of the target coordinate system is the time axis and the vertical axis is the position. For example, Figure 2 The two curves shown are the first trajectory and the second trajectory respectively, where Figure 2 The first trajectory shown is the first trajectory before alignment. The calibration device moves the first trajectory along the time axis by a target time to align the first trajectory with the second trajectory. For example, Figure 3 Shown is after moving the first trajectory in Figure 2 by the target time, the schematic diagram of the alignment of the first trajectory and the second trajectory, that is, Figure 3 The first trajectory shown is the first trajectory after alignment.

[0087] As an alternative implementation, during the execution of step 104, the calibration device performs the following steps:

[0088] 301. Obtain a candidate delay time.

[0089] In the embodiments of the present application, the candidate delay time can be regarded as an imaginary value of the imaging delay time of the imaging device.

[0090] In one implementation of obtaining the candidate delay time, the calibration device receives the candidate delay time input by the user through the input component.

[0091] In another implementation of obtaining the candidate delay time, the calibration device receives the candidate delay time sent by the terminal.

[0092] 302. Add the candidate delay time to the timestamps of the trajectory points of the first trajectory to obtain a third trajectory.

[0093] By executing step 302, the calibration device is equivalent to moving the first trajectory along the time axis by the candidate delay time to obtain the third trajectory. For example, the first trajectory includes trajectory point a and trajectory point b. Among them, the timestamp of trajectory point a is t1, the position of trajectory point a is p1, the timestamp of trajectory point b is t2, and the position of trajectory point b is p2. If the candidate delay time is t3, then after the calibration device adds the candidate delay time to the timestamp of trajectory point a, the timestamp of trajectory point a is t1 + t3. At this time, the position of trajectory point a at the moment of (t1 + t3) is p1. Similarly, after the calibration device adds the candidate delay time to the timestamp of trajectory point b, trajectory point b is at the position of p2 at the moment of (t2 + t3).

[0094] 303. When the second trajectory is aligned with the third trajectory, determine that the imaging delay time of the imaging device is the candidate delay time.

[0095] The alignment of the second trajectory and the third trajectory indicates that the time difference between the first trajectory and the second trajectory is the candidate delay time, that is, the imaging delay time of the imaging device is the candidate delay time. Therefore, when the second trajectory is aligned with the third trajectory, the calibration device determines that the imaging delay time of the imaging device is the candidate delay time.

[0096] In this implementation, when the calibration device obtains the candidate delay time, it obtains the third trajectory by adding the candidate delay time to the timestamp of the second trajectory. When the second trajectory is aligned with the third trajectory, the imaging delay time of the imaging device can be determined.

[0097] As an alternative implementation, the calibration device determines the alignment of the second trajectory and the third trajectory by performing the following steps: 401. When the difference between the second trajectory and the third trajectory is less than or equal to the first threshold, it is determined that the second trajectory and the third trajectory are aligned.

[0098] In a possible implementation manner, the difference between the second trajectory and the third trajectory may be the average value of the position differences at the same time. For example, the second trajectory includes trajectory points a and b, and the third trajectory includes trajectory points c and d. Among them, the timestamps of trajectory point a and trajectory point c are both t1, and the timestamps of trajectory point b and trajectory point d are both t2. If the position of trajectory point a is p1, the position of trajectory point b is p2, the position of trajectory point c is p3, and the position of trajectory point d is p4, then the difference between the second trajectory and the third trajectory is

[0099] In another possible implementation manner, the difference between the second trajectory and the third trajectory may be the average value of the time differences at the same position. For example, the second trajectory includes trajectory points a and b, and the third trajectory includes trajectory points c and d. Among them, the positions of trajectory point a and trajectory point c are both p1, and the positions of trajectory point b and trajectory point d are both p2. If the timestamp of trajectory point a is t1, the timestamp of trajectory point b is t2, the timestamp of trajectory point c is t3, and the timestamp of trajectory point d is t4, then the difference between the second trajectory and the third trajectory is

[0100] A large difference between the second trajectory and the third trajectory indicates that the second trajectory and the third trajectory are not aligned, and a small difference between the second trajectory and the third trajectory indicates that the second trajectory and the third trajectory are aligned. In the embodiments of the present application, the calibration device determines whether the difference between the second trajectory and the third trajectory is large or small based on the first threshold. Specifically, when the difference between the second trajectory and the third trajectory is less than or equal to the first threshold, it indicates that the difference between the second trajectory and the third trajectory is small, and thus it is determined that the second trajectory and the third trajectory are aligned. When the difference between the second trajectory and the third trajectory is greater than the first threshold, it indicates that the difference between the second trajectory and the third trajectory is large, and thus it is determined that the second trajectory and the third trajectory are not aligned.

[0101] As an alternative implementation, the second trajectory includes a first inflection point, the trajectory points of the second trajectory other than the first inflection point are first non - inflection points, the third trajectory includes a second inflection point, and the trajectory points of the third trajectory other than the second inflection point are second non - inflection points.

[0102] The calibration device determines the alignment of the second trajectory and the third trajectory by performing the following steps:

[0103] 501. When the difference between the first inflection point and the second inflection point is less than or equal to the second threshold, and the difference between the first non - inflection point and the second non - inflection point is less than or equal to the third threshold, it is determined that the second trajectory is aligned with the third trajectory.

[0104] The first inflection point represents the position where the movement direction of the imaging device flips, and the second inflection point represents the position where the movement direction of the reference object flips. Therefore, the difference between the first inflection point and the second inflection point can represent the matching degree of the movement trend of the imaging device and the movement trend of the reference object. Specifically, the smaller the difference between the first inflection point and the second inflection point, the higher the matching degree of the movement trend of the imaging device and the movement trend of the reference object, that is, the movement trend of the first trajectory and the movement trend of the second trajectory have a high matching degree.

[0105] Since errors may exist in both the first trajectory and the second trajectory, to improve the accuracy of judging the alignment of the first trajectory and the second trajectory, the matching degree of the movement trends of the first trajectory and the second trajectory can be used as a more important judgment basis, and the difference between the first non - inflection point of the first trajectory and the second non - inflection point of the second trajectory can be used as a relatively less important judgment basis.

[0106] In the embodiment of the present application, the second threshold is the basis for judging whether the difference between the first inflection point and the second inflection point is large or small. Specifically, when the difference between the first inflection point and the second inflection point is less than or equal to the second threshold, it indicates that the difference between the first inflection point and the second inflection point is small; when the difference between the first inflection point and the second inflection point is greater than the second threshold, it indicates that the difference between the first inflection point and the second inflection point is large.

[0107] In the embodiment of the present application, the third threshold is the basis for judging whether the difference between the first non - inflection point and the second non - inflection point is large or small. Specifically, when the difference between the first non - inflection point and the second non - inflection point is less than or equal to the third threshold, it indicates that the difference between the first non - inflection point and the second non - inflection point is small; when the difference between the first non - inflection point and the second non - inflection point is greater than the third threshold, it indicates that the difference between the first non - inflection point and the second non - inflection point is large.

[0108] Since the matching degree of the movement trends of the first trajectory and the second trajectory is a more important judgment basis, and the difference between the first non - inflection point of the first trajectory and the second non - inflection point of the second trajectory is a relatively less important judgment basis, and the third threshold is smaller than the second threshold. Therefore, the calibration device determines that the second trajectory is aligned with the third trajectory when the difference between the first inflection point and the second inflection point is less than or equal to the second threshold, and the difference between the first non - inflection point and the second non - inflection point is less than or equal to the third threshold. This can improve the accuracy of judging the alignment of the first trajectory and the second trajectory.

[0109] As an alternative embodiment, the calibration device determines the position of the reference object in the target image sequence by performing the following steps:

[0110] 601. Segment the images in the above-mentioned target image sequence to obtain the segmentation result sequence of the above-mentioned target image sequence.

[0111] In the embodiments of the present application, the segmentation process is used to determine the reference object from the image. Specifically, the calibration device can determine the pixels with the semantics of the reference object in the image by segmenting the image, and then use the pixel region including the pixels with the semantics of the reference object as the pixel region covered by the reference object. The pixel region covered by the reference object is the segmentation result of the segmentation process.

[0112] By segmenting one image in the target image sequence, the calibration device can obtain a segmentation result. By segmenting each image in the target image sequence respectively, a segmentation result sequence can be obtained. For example, if the target image sequence includes image a and image b, the calibration device obtains segmentation result c by segmenting image a and obtains segmentation result d by segmenting image b. At this time, the segmentation result sequence includes segmentation result c and segmentation result d.

[0113] 602. Determine the position of the above-mentioned reference object in the above-mentioned target image sequence according to the above-mentioned segmentation result sequence.

[0114] The segmentation results in the segmentation result sequence are the reference objects. Therefore, the calibration device can determine the position of the segmentation results in the target image sequence according to the segmentation result sequence, and obtain the position of the reference object in the target image sequence.

[0115] As an alternative implementation, the calibration device performs the following steps during the execution of step 602:

[0116] 701. Obtain the position sequence of the above-mentioned reference object according to the position of the above-mentioned reference object in the above-mentioned target image sequence.

[0117] Based on the position of the reference object in the target image sequence, the calibration device can determine the position of the reference object in each image of the target image sequence, and then obtain the position sequence of the reference object. For example, the target image sequence includes image a and image b, where the timestamp of image a is t1 and the timestamp of image b is t2. The position of the reference object in image a is p1 and the position of the reference object in image b is p2. Then the position sequence of the reference object is that the position of the reference object at time t1 is p1 and the position of the reference object at time t2 is p2.

[0118] 702. Perform curve fitting on the above-mentioned position sequence of the reference object to obtain the above-mentioned second trajectory.

[0119] By performing curve fitting on the position sequence of the reference object, the calibration device can convert the discrete positions in the position sequence into a continuous trajectory curve, which is the second trajectory.

[0120] In this embodiment, when the calibration device obtains the position sequence of the reference object according to the position of the reference object in the target image sequence, the second trajectory can be obtained by performing curve fitting on the position sequence of the reference object.

[0121] As an alternative embodiment, after obtaining the imaging delay time of the imaging device, the calibration device further performs the following steps:

[0122] 801. Obtain the image to be processed and the target conversion relationship collected by the above imaging device.

[0123] In the embodiments of the present application, the image to be processed includes a target object, where the target object can be any object. For example, the target object is a kidney, or the target object is a kidney stone. The timestamp of the image to be processed is the first time, that is, the image to be processed is an image generated at the first time.

[0124] In the embodiments of the present application, the target conversion relationship is the conversion relationship between the pixel coordinate system and the world coordinate system of the image to be processed, that is, according to the target conversion relationship, the coordinates of any pixel in the image to be processed in the world coordinate system can be determined. For example, the image to be processed includes pixel a, and the object point corresponding to pixel a in the real world is object point b. Then, according to the target conversion relationship and the position of pixel a in the pixel coordinate system of the image to be processed, the coordinates of object point b in the world coordinate system can be determined. Optionally, the target conversion relationship is the homography matrix between the pixel coordinate system and the world coordinate system of the image to be processed.

[0125] In one implementation of obtaining the image to be processed, there is a communication connection between the calibration device and the ultrasonic probe, and the calibration device obtains the ultrasonic image collected by the ultrasonic probe as the image to be processed through this communication connection.

[0126] In another implementation of obtaining the image to be processed, the calibration device receives the image to be processed input by the user through the input component.

[0127] In yet another implementation of obtaining the image to be processed, the calibration device obtains the image to be processed by receiving the image to be processed sent by the terminal.

[0128] In one implementation of obtaining the target conversion relationship, the calibration device receives the target conversion relationship input by the user through the input component.

[0129] In another implementation of obtaining the target conversion relationship, the calibration device obtains the target conversion relationship by receiving the target conversion relationship sent by the terminal.

[0130] It should be understood that in the embodiments of the present application, the steps of the calibration device for acquiring the image to be processed and acquiring the target conversion relationship may be executed separately or simultaneously, and the present application does not limit this.

[0131] 802. According to the position of the above target object in the above image to be processed and the above target conversion relationship, obtain the target position of the above target object in the above world coordinate system as the position of the above target object body at the second time.

[0132] In the embodiments of the present application, the second time is the time when the imaging device starts to acquire the image to be processed. Due to the imaging delay of the imaging device, after the imaging device starts to acquire the image to be processed and after the imaging delay time, the imaging device generates the image to be processed, that is, the imaging device generates the image to be processed at the first time. That is to say, the second time is the difference between the first time and the imaging delay time. For example, the first time is 17:48:03.050 on December 25, 2022, and the imaging delay time of the imaging device is 40 milliseconds, then the second time is 40 milliseconds before the first time, that is, 17:48:03.010 on December 25, 2022.

[0133] Since the time when the imaging device acquires the image to be processed is the second time, the position of the target object in the image to be processed corresponds to the position of the target object in the world coordinate system when the imaging device acquires the image to be processed. Therefore, the calibration device can obtain the position of the target object in the world coordinate system at the second time according to the position of the target object in the image to be processed and the target conversion relationship.

[0134] In this embodiment, when the calibration device acquires the image to be processed including the target object and the target conversion relationship, it can obtain the target position of the target object in the world coordinate system when the imaging device acquires the image to be processed according to the position of the target object in the image to be processed and the target conversion relationship, thereby realizing the correction of the position of the target object in the world coordinate system.

[0135] Based on the technical solution provided by the embodiments of the present application, the embodiments of the present application also provide a possible application scenario.

[0136] With the progress of medical imaging technology, ultrasound image-guided percutaneous puncture has been widely used. In the traditional diagnosis method, the doctor holds the probe and observes the lesion on the two-dimensional ultrasound image from any angle. The doctor judges the appropriate puncture angle and puncture depth based on experience. For example, in the clinical puncture operation of percutaneous nephrolithotripsy, the doctor can observe the process of the puncture operation through the ultrasound image and judge whether the puncture is appropriate.

[0137] To better observe the situation of the puncture surgery, many current methods perform ultrasonic three-dimensional reconstruction based on two-dimensional ultrasonic images to obtain three-dimensional ultrasonic images, and then observe the puncture surgery through the three-dimensional ultrasonic images. However, since the ultrasonic probe for collecting two-dimensional ultrasonic images has a fixed imaging refresh rate, the imaging of two-dimensional ultrasonic images has the characteristic of delay. For example, if the imaging refresh rate of the ultrasonic probe is 30 FPS (frames per second), then the imaging delay time of the ultrasonic probe is 1 / 30 second. Another example is that when the ultrasonic probe is in the color flow mapping (CFM) mode and the imaging refresh rate of the ultrasonic probe drops from 30 FPS to 15 FPS, the imaging delay time of the ultrasonic probe will change from 1 / 30 second to 1 / 15 second.

[0138] Obviously, the imaging delay of two-dimensional ultrasonic images will introduce errors into ultrasonic three-dimensional reconstruction. Specifically, during the percutaneous puncture surgery using ultrasonic three-dimensional reconstruction, three-dimensional ultrasonic images are reconstructed based on the two-dimensional ultrasonic images collected by the ultrasonic probe. However, due to the delay characteristic of the imaging of the ultrasonic probe, there is a time difference between the timestamp of the two-dimensional ultrasonic image and the time of collecting the two-dimensional ultrasonic image during the process of collecting the two-dimensional ultrasonic image by the ultrasonic probe, which leads to errors in ultrasonic three-dimensional reconstruction. Therefore, how to correct the acquisition time of two-dimensional ultrasonic images based on the generation time (i.e., the timestamp) of two-dimensional ultrasonic images is of great significance for improving the accuracy of ultrasonic three-dimensional reconstruction.

[0139] Based on the technical solution provided in the embodiments of the present application, the imaging delay time of the ultrasonic probe can be calibrated, and then the acquisition time of two-dimensional ultrasonic images can be corrected based on this imaging delay time. Specifically, a device for realizing optical tracking or a device for magnetic positioning is fixed on the ultrasonic probe. The ultrasonic probe is reciprocated vertically along the bottom of the water tank, where the bottom of the water tank includes a strong echo line, or the ultrasonic probe is reciprocated horizontally along the nylon line of the phantom. At this time, the strong echo line at the bottom of the water tank or the nylon line of the phantom is the above-mentioned reference object.

[0140] Record the position of the ultrasonic probe in the world coordinate system obtained by optical tracking or magnetic positioning during the movement of the ultrasonic probe, and save each frame of two-dimensional ultrasonic image collected by the ultrasonic probe. Use an image segmentation algorithm to identify the strong echo line at the bottom of the water tank or the nylon line of the phantom from each frame of ultrasonic image, where Figure 4 the line in is the strong echo line at the bottom of the water tank, Figure 5 the line in is the nylon line of the phantom.

[0141] Determine the first trajectory based on the position of the ultrasound probe in the world coordinate system, determine the second trajectory based on the position of the reference object (strong echo line or phantom nylon line) in the two-dimensional ultrasound image, and display the first trajectory and the second trajectory in the target coordinate system. For details, please refer to Figure 2 . Perform phase registration on the first trajectory and the second trajectory to align the first trajectory with the second trajectory, and obtain the time offset. The state after the alignment of the first trajectory and the second trajectory can be seen in Figure 3 . At this time, this time offset is the imaging delay time of the ultrasound probe.

[0142] After obtaining the imaging delay time of the ultrasound probe, the time stamp of the two-dimensional ultrasound image can be corrected based on this imaging delay time to obtain the time when the two-dimensional ultrasound image is acquired. Then, perform three-dimensional ultrasound reconstruction based on the time when the two-dimensional ultrasound image is acquired, which can improve the accuracy of three-dimensional ultrasound reconstruction.

[0143] Those skilled in the art can understand that in the above method of the specific implementation manner, the writing order of each step does not mean a strict execution order and does not constitute any limitation to the implementation process. The specific execution order of each step should be determined according to its function and possible internal logic.

[0144] If the technical solution of this application involves personal information, before the product applying the technical solution of this application processes personal information, it has clearly informed the personal information processing rules and obtained the personal's autonomous consent. If the technical solution of this application involves sensitive personal information, before the product applying the technical solution of this application processes sensitive personal information, it has obtained the personal's separate consent and at the same time meets the requirement of "express consent". For example, at a personal information collection device such as a camera, a clear and prominent sign is set to inform that the personal information collection range has been entered and personal information will be collected. If the personal voluntarily enters the collection range, it is regarded as consenting to the collection of their personal information; or on the device for processing personal information, when the personal information processing rules are informed by obvious signs / information, personal authorization is obtained through pop-up information or asking the personal to upload their personal information by themselves, etc.; among them, personal information processing may include information such as personal information processors, personal information processing purposes, processing methods, and types of personal information processed.

[0145] The method of the embodiment of this application is elaborated in detail above, and the device of the embodiment of this application is provided below.

[0146] Please refer to Figure 6 , Figure 6 which is a schematic structural diagram of a calibration device provided by an embodiment of this application. The calibration device 1 includes: an acquisition unit 11, a processing unit 12, and an alignment unit 13. Optionally, the calibration device 1 further includes: a determination unit 14. Specifically:

[0147] An acquisition unit 11, configured to acquire a first trajectory of an imaging device and a target image sequence, where the target image sequence is an image sequence obtained by the imaging device shooting a reference object during the movement along the first trajectory;

[0148] A processing unit 12, configured to obtain a second trajectory of the reference object in the target image sequence according to the position of the reference object in the target image sequence;

[0149] An alignment unit 13, configured to obtain an imaging delay time of the imaging device by aligning the first trajectory with the second trajectory.

[0150] Combined with any embodiment of the present application, the alignment unit 13 is specifically configured to:

[0151] When aligning the first trajectory with the second trajectory by moving a target time along the time axis, determine that the imaging delay time of the imaging device is the target time.

[0152] Combined with any embodiment of the present application, the alignment unit 13 is specifically configured to:

[0153] Obtain a candidate delay time;

[0154] Add the candidate delay time to the time stamps of the trajectory points of the first trajectory to obtain a third trajectory;

[0155] When the second trajectory is aligned with the third trajectory, determine that the imaging delay time of the imaging device is the candidate delay time.

[0156] Combined with any embodiment of the present application, the calibration device 1 further includes:

[0157] A determination unit 14, configured to determine that the second trajectory is aligned with the third trajectory when the difference between the second trajectory and the third trajectory is less than or equal to a first threshold.

[0158] Combined with any embodiment of the present application, the second trajectory includes a first inflection point, the trajectory points of the second trajectory except the first inflection point are first non - inflection points, the third trajectory includes a second inflection point, and the trajectory points of the third trajectory except the second inflection point are second non - inflection points;

[0159] The determination unit 14 is further configured to determine that the second trajectory is aligned with the third trajectory when the difference between the first inflection point and the second inflection point is less than or equal to a second threshold and the difference between the first non - inflection points and the second non - inflection points is less than or equal to a third threshold, where the second threshold is less than the third threshold.

[0160] In combination with any embodiment of the present application, the processing unit 12 is further configured to:

[0161] Perform segmentation processing on the images in the target image sequence to obtain a segmentation result sequence of the target image sequence;

[0162] Determine the position of the reference object in the target image sequence according to the segmentation result sequence.

[0163] In combination with any embodiment of the present application, the processing unit 12 is specifically configured to:

[0164] Obtain a position sequence of the reference object according to the position of the reference object in the target image sequence;

[0165] Perform curve fitting on the position sequence of the reference object to obtain the second trajectory.

[0166] In combination with any embodiment of the present application, the acquisition unit 11 is further configured to acquire a to-be-processed image and a target conversion relationship collected by the imaging device, where the to-be-processed image includes a target object, the time stamp of the to-be-processed image is the first time, and the target conversion relationship is the conversion relationship between the pixel coordinate system and the world coordinate system of the to-be-processed image;

[0167] The processing unit 12 is further configured to obtain the target position of the target object in the world coordinate system according to the position of the target object in the to-be-processed image and the target conversion relationship, and use it as the position of the target object body at the second time, where the second time is the difference between the first time and the imaging delay time.

[0168] In the embodiments of the present application, the calibration device acquires the first trajectory of the imaging device and the target image sequence, where the target image sequence is an image sequence obtained by the imaging device shooting a reference object during the movement along the first trajectory. Then, according to the position of the reference object in the target image sequence, the second trajectory of the reference object in the target image sequence can be obtained. Since there is relative movement between the imaging device and the reference object during the acquisition of the target image sequence by the imaging device, the imaging delay time of the imaging device can be obtained by aligning the first trajectory and the second trajectory.

[0169] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0170] Figure 7Schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 further includes an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23, and the output device 24 are coupled through a connector, which includes various interfaces, transmission lines, buses, etc., and the embodiments of the present application do not limit this. It should be understood that in various embodiments of the present application, coupling refers to the mutual connection in a specific manner, including direct connection or indirect connection through other devices. For example, they can be connected through various interfaces, transmission lines, buses, etc.

[0171] The processor 21 can be one or more graphics processing units (GPUs). When the processor 21 is a single GPU, the GPU can be a single-core GPU or a multi-core GPU. Optionally, the processor 21 can be a processor group composed of multiple GPUs, and multiple processors are coupled to each other through one or more buses. Optionally, the processor can also be other types of processors, etc., and the embodiments of the present application do not limit this.

[0172] The memory 22 can be used to store computer program instructions and various computer program codes including the program codes for executing the solution of the present application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and this memory is used for relevant instructions and data.

[0173] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0174] It can be understood that in the embodiments of the present application, the memory 22 can not only be used to store relevant instructions, but also be used to store relevant data, and the embodiments of the present application do not limit the specific data stored in this memory.

[0175] It can be understood that Figure 7Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also separately include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0176] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0177] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that each embodiment of the present application has different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, parts not described or not described in detail in a certain embodiment can be referred to the descriptions of other embodiments.

[0178] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in an electrical, mechanical, or other form.

[0179] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, the functional units in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0181] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital versatile disc (DVD)), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0182] Those of ordinary skill in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing relevant hardware with a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The foregoing storage medium includes: various media that can store program codes such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

Claims

1. A calibration method, characterized in that, The method includes: Obtaining a first trajectory of an imaging device and a target image sequence, where the target image sequence is an image sequence obtained by the imaging device capturing a reference object during the movement according to the first trajectory; Obtaining a second trajectory of the reference object in the target image sequence according to the position of the reference object in the target image sequence; Obtaining the imaging delay time of the imaging device by aligning the first trajectory with the second trajectory; the step of obtaining the imaging delay time of the imaging device by aligning the first trajectory with the second trajectory includes: when aligning the first trajectory with the second trajectory by moving a target time along the time axis, determining the imaging delay time of the imaging device as the target time; Obtaining a to-be-processed image collected by the imaging device and a target conversion relationship, where the to-be-processed image includes a target object, the timestamp of the to-be-processed image is a first time, and the target conversion relationship is a conversion relationship between the pixel coordinate system and the world coordinate system of the to-be-processed image; Obtaining a target position of the target object in the world coordinate system according to the position of the target object in the to-be-processed image and the target conversion relationship, and using it as the position of the target object at a second time, where the second time is the difference between the first time and the imaging delay time.

2. The method according to claim 1, wherein The step of obtaining the imaging delay time of the imaging device by aligning the first trajectory with the second trajectory includes: Obtaining a candidate delay time; Adding the candidate delay time to the timestamps of the trajectory points of the first trajectory to obtain a third trajectory; When the second trajectory is aligned with the third trajectory, determining the imaging delay time of the imaging device as the candidate delay time.

3. The method according to claim 2, wherein Before determining the imaging delay time of the imaging device as the candidate delay time when the second trajectory is aligned with the third trajectory, the method further includes: When the difference between the second trajectory and the third trajectory is less than or equal to a first threshold, determining that the second trajectory is aligned with the third trajectory.

4. The method according to claim 2, characterized in that The second trajectory includes a first inflection point, and the trajectory points of the second trajectory except the first inflection point are first non-inflection points; the third trajectory includes a second inflection point, and the trajectory points of the third trajectory except the second inflection point are second non-inflection points; Before determining the imaging delay time of the imaging device as the candidate delay time when the second trajectory is aligned with the third trajectory, the method further includes: When the difference between the first inflection point and the second inflection point is less than or equal to a second threshold and the difference between the first non-inflection points and the second non-inflection points is less than or equal to a third threshold, determining that the second trajectory is aligned with the third trajectory, where the second threshold is less than the third threshold.

5. The method according to claim 1, characterized in that, Before obtaining the second trajectory of the reference object in the target image sequence according to the position of the reference object in the target image sequence, the method further includes: Performing segmentation processing on the images in the target image sequence to obtain a segmentation result sequence of the target image sequence; Determine the position of the reference object in the target image sequence according to the segmentation result sequence.

6. The method according to claim 1, wherein The obtaining the second trajectory of the reference object in the target image sequence according to the position of the reference object in the target image sequence includes: Obtain the position sequence of the reference object according to the position of the reference object in the target image sequence; Perform curve fitting on the position sequence of the reference object to obtain the second trajectory.

7. A calibration device, characterized in that, The apparatus includes: An obtaining unit, configured to obtain a first trajectory of an imaging device and a target image sequence, where the target image sequence is an image sequence obtained by the imaging device capturing a reference object during the movement according to the first trajectory; A processing unit, configured to obtain a second trajectory of the reference object in the target image sequence according to the position of the reference object in the target image sequence; An alignment unit, configured to obtain the imaging delay time of the imaging device by aligning the first trajectory with the second trajectory; the obtaining the imaging delay time of the imaging device by aligning the first trajectory with the second trajectory includes: determining that the imaging delay time of the imaging device is the target time when the first trajectory is aligned with the second trajectory by moving a target time along the time axis; The obtaining unit is further configured to obtain a to-be-processed image collected by the imaging device and a target conversion relationship, where the to-be-processed image includes a target object, a time stamp of the to-be-processed image is a first time, and the target conversion relationship is a conversion relationship between a pixel coordinate system and a world coordinate system of the to-be-processed image; The processing unit is further configured to obtain a target position of the target object in the world coordinate system according to the position of the target object in the to-be-processed image and the target conversion relationship, and use the target position as the position of the target object at a second time, where the second time is a difference between the first time and the imaging delay time.

8. An electronic device, characterized in that, Includes: A processor and a memory, where the memory is configured to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, A computer program is stored in the computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 6.

Images