Image display method, device, USB camera and medium

Abstract

The embodiments of the present application provide an image display method, device, USB camera and medium. The USB camera is deployed with an ITOF module, and the method includes: acquiring a current image pair by performing a pairing task at a frequency corresponding to a first frame rate, wherein the current image pair includes a current frame RGB image and a target depth image matching the current frame RGB image. According to a pre-calibrated cropping parameter, the target image in the current image pair is cropped to acquire a processed current image pair so that the field of view of the two images in the processed current image pair is the same. If the host computer requests to acquire the RGB image and the depth image, the processed current image pair is sent to the host computer according to the preset frame structure, so that the host computer displays the processed current image pair. The technical solution ensures the effect of synchronously displaying the RGB image and the depth image in the host computer, saving equipment costs.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image display method, device, USB camera and medium. Background Art

[0002] The host computer is the control center of the USB camera. It can view the RGB images collected by the USB camera and control the USB camera through the human-computer interaction interface.

[0003] In some scenarios, the host computer needs to obtain not only the RGB image but also the depth image. At present, in order to meet the above purpose, it is usually necessary to set up a depth data acquisition device to supplement the depth image required by the host computer.

[0004] However, when the prior art displays the RGB image and the depth image synchronously, the display effect is poor and the cost is high. Summary of the invention

[0005] The embodiments of the present application provide an image display method, an apparatus, a USB camera and a medium to improve display effects and reduce equipment costs.

[0006] In a first aspect, an embodiment of the present application provides an image display method, which is applied to a USB camera, wherein the USB camera is equipped with an ITOF module, and the USB camera is used to capture an RGB image at a first frame rate and a depth image at a second frame rate, wherein the first frame rate is greater than or equal to the second frame rate, and the method includes:

[0007] Perform a pairing task at a frequency corresponding to the first frame rate to obtain a current image pair, where the current image pair includes a current frame RGB image and a target depth image matching the current frame RGB image;

[0008] Cropping the target image in the current image pair according to pre-calibrated cropping parameters to obtain a processed current image pair, so that the field of view angles of the two images in the processed current image pair are the same;

[0009] If the host computer requests to obtain the RGB image and the depth image, the processed current image pair is sent to the host computer according to the preset frame structure, so that the host computer displays the processed current image pair, and the preset frame structure includes a first frame header and a first frame footer corresponding to the RGB image, and a second frame header and a second frame footer corresponding to the depth image;

[0010] Among them, the pairing task is: add 1 to i as the new i, if the new i is less than a preset value, determine the current frame depth image as the target depth image; if the new i is greater than or equal to the preset value, determine the previous frame depth image as the target depth image, and reset the new i to zero; the preset value is the ratio of the first frame rate to the preset frequency difference, the preset frequency difference is the difference between the first frame rate and the second frame rate, and the preset value is greater than 1.

[0011] In a second aspect, an embodiment of the present application provides an image display device, which is applied to a USB camera, wherein the USB camera is equipped with an ITOF module, and the USB camera is used to capture an RGB image at a first frame rate and a depth image at a second frame rate, wherein the first frame rate is greater than or equal to the second frame rate, and the device includes:

[0012] A task execution module, configured to execute a pairing task at a frequency corresponding to the first frame rate to obtain a current image pair, wherein the current image pair includes a current frame RGB image and a target depth image matching the current frame RGB image;

[0013] A cropping module, used for cropping the target image in the current image pair according to a pre-calibrated cropping parameter to obtain a processed current image pair so that the field of view angles of the two images in the processed current image pair are the same;

[0014] A sending module, for sending the processed current image pair to the host computer according to a preset frame structure if the host computer requests to obtain the RGB image and the depth image, so that the host computer displays the processed current image pair, wherein the preset frame structure includes a first frame header and a first frame footer corresponding to the RGB image, and a second frame header and a second frame footer corresponding to the depth image;

[0015] Among them, the pairing task is: add 1 to i as the new i, if the new i is less than a preset value, determine the current frame depth image as the target depth image; if the new i is greater than or equal to the preset value, determine the previous frame depth image as the target depth image, and reset the new i to zero; the preset value is the ratio of the first frame rate to the preset frequency difference, the preset frequency difference is the difference between the first frame rate and the second frame rate, and the preset value is greater than 1.

[0016] In a third aspect, an embodiment of the present application provides a USB camera, comprising:

[0017] Memory, processor and ITOF module;

[0018] The ITOF module is used to collect depth images;

[0019] The memory stores computer-executable instructions;

[0020] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the implementation of the first aspect as described above.

[0021] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the implementation method of the first aspect as above.

[0022] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the implementation method of the first aspect as above.

[0023] The image display method, device, USB camera and medium provided by the embodiment of the present application, the USB camera is deployed with an ITOF module, and the current image pair is obtained by performing the pairing task at a frequency corresponding to the first frame rate, and the current image pair includes a current frame RGB image and a target depth image matching the current frame RGB image. According to the pre-calibrated cropping parameters, the target image in the current image pair is cropped to obtain the processed current image pair so that the field of view of the two images in the processed current image pair is the same. If the host computer requests to obtain the RGB image and the depth image, the processed current image pair is sent to the host computer according to the preset frame structure, so that the host computer displays the processed current image pair. In this technical solution, by performing the pairing task at a frequency corresponding to the first frame rate, the output frame rates of the RGB image and the depth image can be adjusted to be consistent to ensure that the two can maintain time consistency when displayed on the host computer. Furthermore, the target image in the current image pair is cropped based on the cropping parameters so that the two images in the processed current image pair have the same field of view; finally, the two images in the processed current image pair are simultaneously transmitted to the host computer through a preset frame structure for display. This solution ensures the synchronous display of the RGB image and the depth image in the host computer by adjusting the output frame rate and field of view of the RGB image and the depth image, saving equipment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0025] Figure 1 A schematic diagram of a scene of an image display method provided in an embodiment of the present application;

[0026] Figure 2 Schematic diagram of the process of the image display method provided in the embodiment of the present application Figure 1 ;

[0027] Figure 3 A schematic diagram of the image field of view provided in an embodiment of the present application;

[0028] Figure 4 Schematic diagram of image cropping provided in the embodiment of the present application Figure 1 ;

[0029] Figure 5 Schematic diagram of image cropping provided in the embodiment of the present application Figure 2 ;

[0030] Figure 6 Schematic diagram of RGB image and depth image provided in the embodiment of the present application;

[0031] Figure 7 Schematic diagram of the process of the image display method provided in the embodiment of the present application Figure 2 ;

[0032] Figure 8 A schematic diagram of a calibrated RGB image in an ideal scenario provided in an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of a calibrated RGB image in an actual scenario provided in an embodiment of the present application;

[0034] Fig.10 Schematic diagram of the process of the image display method provided in the embodiment of the present application Figure 3 ;

[0035] Fig.11 A schematic diagram of a new rectangle provided for an embodiment of the present application;

[0036] Fig.12 A schematic diagram of a corrected calibrated RGB image provided in an embodiment of the present application;

[0037] Fig.13 A schematic diagram of a test environment provided for an embodiment of the present application;

[0038] Fig.14 Schematic diagram of the process of the image display method provided in the embodiment of the present application Figure 4 ;

[0039] Fig.15 Schematic diagram of the process of the image display method provided in the embodiment of the present application Figure 5 ;

[0040] Fig.16 A schematic diagram of the structure of the image display device provided in this application;

[0041] Fig.17This is a schematic diagram of the structure of the USB camera provided in this application.

[0042] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0043] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0044] First, the terms used in this application are explained:

[0045] Indirect Time-of-Flight (ITOF) module: A depth data acquisition device, including an acquisition end and a transmitter, for acquiring depth data. Specifically, the transmitter emits a modulated (for example, by changing the intensity and frequency of the light) light signal to the surface of the target object. When the light signal reaches the surface of the target object, it will be reflected. The acquisition end then receives the reflected light signal and can calculate the depth data of the target object based on the time difference between the transmitted light signal and the received reflected light signal.

[0046] Red-Green-Blue Image (RGB image): is an image representation based on the RGB color model. The RGB color model is an additive color model that uses red, green, and blue as primary colors. In this model, a variety of colors can be produced by mixing red, green, and blue with different intensities. Each primary color has an intensity value, which usually ranges from 0 to 255 (in the case of 8-bit representation).

[0047] Universal Serial Bus (USB) camera: A camera that is based on the USB transmission protocol and uses a USB interface for communication.

[0048] Data preview host computer (referred to as host computer): application software installed on the Windows host platform for USB camera preview.

[0049] Next, the background technology involved in this application is explained:

[0050] The host computer has a human-computer interaction interface. After the host computer and the USB camera are connected in communication, the operator can view the video captured by the USB camera through the human-computer interaction interface, and can also input control instructions (for example, start shooting, stop shooting, etc.) through the human-computer interaction interface to control the USB camera. In other words, the host computer is the control center of the USB camera.

[0051] In some scenarios, the host computer needs to obtain the RGB image and depth image of the target object at the same time. For example, when making an electronic class card, in order to realize the attendance management function of the electronic class card, it is necessary to create a facial feature library of class members in advance so that subsequent comparisons can be made based on the facial feature library. When creating the facial feature library of class members, it is necessary to use a USB camera to collect the RGB images of the faces of class members, and use a depth data acquisition device to collect the depth images of the faces of class members, so as to extract the facial features of class members from the RGB images and the depth images of the faces.

[0052] However, the prior art has the following technical problems:

[0053] 1. Since the RGB image and the depth image are collected by different devices and the two devices need to be installed in different positions, there is a difference in the field of view between the RGB image and the depth image. The effect of synchronously displaying the RGB image and the depth image in the host computer is poor.

[0054] 2. The two devices are of different types, so the frame rates of the output RGB images and depth images are also different, which causes the RGB images and depth images to be out of sync in time, further resulting in poor results in the synchronous display of the RGB images and depth images in the host computer.

[0055] 3. Using two independent devices to collect RGB images and depth images is costly.

[0056] That is to say, the prior art has the problems of poor effect and high cost in synchronously displaying RGB images and depth images.

[0057] Based on the above technical problems, the technical concept of the present application is as follows: an ITOF module can be deployed in a USB camera so that the USB camera has the function of simultaneously capturing RGB images and depth images, thereby reducing costs. At the same time, before outputting the RGB image and the depth image, the RGB image and the depth image can be paired according to the output frame rate of the RGB image and the depth image to achieve synchronous output of the RGB image and the depth image. Furthermore, the USB camera can be calibrated in advance to determine the cropping parameters. After pairing the RGB image and the depth image, the RGB image or the depth image can also be cropped to ensure that the field of view of the two is the same, thereby improving the effect of the host computer on the synchronous display of the RGB image and the depth image.

[0058] Figure 1 A schematic diagram of a scene of an image display method provided in an embodiment of the present application. Figure 1 As shown, the scene includes a USB camera and a host computer, and the USB camera is deployed with an ITOF module.

[0059] In actual applications, when the operator wants to view the RGB image and depth image of the target object, the operator can control the USB camera through the human-computer interaction interface. The USB camera responds to the user's control operation on the USB camera and sends a control instruction to the USB camera through the USB camera controller. After receiving the control instruction sent by the host computer, the USB camera continuously collects the RGB image and depth image of the target object, and matches the RGB image and the depth image to obtain the current image pair. After that, the RGB image or the depth image in the current image pair is cropped according to the cropping parameter to generate the processed current image pair. Finally, the processed current image pair is encoded according to the preset frame structure, and image data is generated and sent to the host computer. After receiving the image data, the host computer decodes the image data and displays the RGB image and depth image contained in the processed current image pair through the human-computer interaction interface so that the operator can preview it.

[0060] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0061] Figure 2 Schematic diagram of the process of the image display method provided in the embodiment of the present application Figure 1 .like Figure 2 As shown, the image display method can be implemented by the following steps:

[0062] S21. Execute a pairing task according to a frequency corresponding to the first frame rate to obtain a current image pair.

[0063] The execution subject of the embodiment of the present application is a USB camera, in which an ITOF module is deployed. The USB camera can capture RGB images at a first frame rate and capture depth images at a second frame rate, and the first frame rate is greater than or equal to the second frame rate.

[0064] It should be understood that the frequency corresponding to the first frame rate is the reciprocal of the first frame rate. Exemplarily, assuming that the first frame rate is xfps, the frequency corresponding to the first frame rate is 1000 / x milliseconds each time.

[0065] The current image pair includes a current frame RGB image and a target depth image matching the current frame RGB image.

[0066] Specifically, the pairing task is: add 1 to i as the new i, if the new i is less than the preset value, determine the current frame depth image as the target depth image; if the new i is greater than or equal to the preset value, determine the previous frame depth image as the target depth image, and reset the new i to zero. The preset value is the ratio of the first frame rate to the preset frequency difference, the preset frequency difference is the difference between the first frame rate and the second frame rate, and the preset value is greater than 1.

[0067] It should be understood that if the value calculated by x / (xy) is not an integer, the value is rounded down and the value obtained by rounding down is determined as the preset value.

[0068] For example, assume that the acquisition frame rate of the RGB image is x fps, the acquisition frame rate of the depth image is y fps, and x >y. Define the initial value of the parameter (cnt) to be 0, create a pairing task, and execute the pairing task at a frequency of 1000 / x ms each time. Each time the pairing task is executed, obtain the current frame RGB image, and add 1 to the current value (i) of cnt as the new current value (i+1) of cnt. If cnt < x / (xy), obtain the current frame depth image and combine it with the current frame RGB image to generate the current image pair; if cnt ≥ x / (xy), obtain the previous frame depth image and combine it with the current frame RGB image to generate the current image pair.

[0069] For example, assuming that x is 4 and y is 2, the USB camera captures 4 RGB images within 1 second, namely RGB image 1, RGB image 2, RGB image 3, and RGB image 4, and the USB camera captures 2 depth images within 1 second, namely depth image 1 and depth image 2. After performing the pairing task, the current image pairs generated are (RGB image 1, depth image 1), (RGB image 2, depth image 1), (RGB image 3, depth image 2), and (RGB image 4, depth image 2).

[0070] In practical applications, if the preset value is less than or equal to 1, the acquired depth image can be stored in the buffer. When the current frame RGB image is acquired, the depth image with the shortest storage time is extracted from the buffer as the target depth image, which together with the current frame RGB image constitutes the current image pair.

[0071] It can be seen that the pairing task pairs the RGB image and the depth image to adjust the output frame rates of the RGB image and the depth image to be consistent, which can ensure the time consistency and synchronization of the two images in the current image pair and avoid the problem of information mismatch in image display due to different acquisition frame rates.

[0072] S22. Crop the target image in the current image pair according to the pre-calibrated cropping parameters to obtain a processed current image pair, so that the field of view angles of the two images in the processed current image pair are the same.

[0073] The cropping parameters include a target image type, a resizing factor, and a cropping start coordinate, and the target image type is an RGB type or a depth type. It should be understood that an RGB image is an RGB type, and a depth image is a depth type.

[0074] It should be understood that the cropping parameters are obtained and stored when the USB camera is calibrated. The specific calibration process will be explained in subsequent embodiments and will not be repeated here.

[0075] In practical applications, due to differences in structure, lens, and output format, the field of view of the depth image is smaller than that of the RGB image. Specifically, when the field of view of the depth image is smaller than that of the RGB image, the target image type is the RGB type, that is, the RGB image needs to be cropped; similarly, when the field of view of the RGB image is smaller than that of the depth image, the target image type is the depth type, that is, the depth image needs to be cropped.

[0076] Figure 3 This is a schematic diagram of the image field of view provided in the embodiment of the present application. Figure 3As shown in the figure, assuming that the green part is the field of view of the RGB image and the red part is the field of view of the depth image, it means that the field of view of the depth image is included in the field of view of the RGB image, that is, the field of view of the depth image is smaller than that of the RGB image. Assuming that the blue part is the field of view of the RGB image and the red part is the field of view of the depth image, it means that the field of view of the RGB image is included in the field of view of the depth image, that is, the field of view of the RGB image is smaller than that of the depth image.

[0077] In a specific implementation, the target image in the current image pair can be determined according to the target image type. Then, the target image is cropped according to the cropping start coordinates to crop the target image to the target image size, and the processed current image pair is obtained. The target image size is the product of the initial image size of the non-target image in the current image pair and the size adjustment coefficient.

[0078] It should be understood that the non-target image is another image other than the target image in the current image pair.

[0079] Among them, when cropping, cropping can be performed in a preset direction order, and the preset direction order is pre-set according to the position of the cropping start coordinates in the cropped image. For example, assuming that the cropping start coordinates are the upper left vertex in the cropped image, the preset direction order can be right, down, left, and up; assuming that the cropping start coordinates are the lower right vertex in the cropped image, the preset direction order can be up, left, down, and right.

[0080] Assume that the two opposite vertices of the rectangle (such as the upper left vertex and the lower right vertex) are calibration marks, and the coordinates of the calibration marks in the current RGB image are (r1", g1"), (r4", g4"), and the coordinates of the calibration marks in the target depth image are (t1", f1"), (t4", f4"), and the initial image size of the target depth image is a. b, the initial image size of the current RGB image is c d. When the field of view of the depth image is smaller than that of the RGB image, Figure 4 Explain the scene of image cropping. When the field of view of the RGB image is smaller than the field of view of the depth image, Figure 5 Explain the scene of image cropping.

[0081] Figure 4 Schematic diagram of image cropping provided in the embodiment of the present application Figure 1 .like Figure 4 As shown, the cropping parameters are , the target image is the target RGB image, and the target image size is , = a, = b, the starting coordinates of the clipping (the coordinates of the red point) are (r1"- t1", g1"- f1").

[0082] Figure 5 Schematic diagram of image cropping provided in the embodiment of the present application Figure 2 .like Figure 5 As shown, the cropping parameters are , the target image is the current depth image, and the target image size is , = c, = d, the starting coordinates of cutting are (t1"- r1", f1"- g1").

[0083] In the above technical scheme, the target image to be cropped in the current image pair is determined by the target image type, the position to start cropping in the target image is determined by the cropping start coordinates, and the target image size of the cropped image is determined by the size adjustment coefficient, thereby ensuring the accuracy of the cropping operation and ensuring that the field of view angles of the two images in the current image pair after processing are the same.

[0084] S23: If the host computer requests to obtain the RGB image and the depth image, the processed current image pair is sent to the host computer according to the preset frame structure, so that the host computer displays the processed current image pair.

[0085] In practical applications, the host computer can send a control instruction to the USB camera to obtain the RGB image and / or depth image captured by the USB camera.

[0086] In practical applications, since the current USB camera can only send RGB images to the host computer, the current frame structure needs to be modified to form a preset frame structure, which includes the first frame header and the first frame tail corresponding to the RGB image, and the second frame header and the second frame tail corresponding to the depth image.

[0087] Exemplarily, the preset frame structure can be implemented by the following Table 1.

[0088] Table 1

[0089]

[0090] As shown in Table 1, the frame header FF D8 is used as the first frame header of the RGB image, FF D9 is used as the first frame footer of the RGB image and the second frame header of the depth image, and FF DD is used as the second frame footer of the depth image.

[0091] In one possible implementation, if the host computer requests to obtain the RGB image and the depth image, the RGB image in the processed current image pair is placed between the first frame header and the first frame tail, and the depth image in the processed current image pair is placed between the second frame header and the second frame tail, and the image data is generated and sent to the host computer.

[0092] Optionally, the image data may be sent to a host computer via a USB Video Class (UVC) transmission protocol.

[0093] Correspondingly, for the host computer, after obtaining the image data sent by the USB camera through the UVC transmission protocol, it searches for the first frame header FF D8 and the first frame tail FF D9, takes the data between the first frame header and the first frame tail, and the data is the RGB image. It continues to search for the second frame tail FF DD, takes the data between FF D9 and FF DD, and the data is the depth image.

[0094] It should be understood that in the above technical solution, the RGB image is placed between the first frame header and the first frame tail, and the depth image is placed between the second frame header and the second frame tail. In this way, after the upper computer obtains the image data, the RGB image and the depth image can be extracted from it according to the positions of the first frame header, the first frame tail, the second frame header and the second frame tail.

[0095] Figure 6 Schematic diagram of RGB image and depth image provided in the embodiment of the present application. Figure 6 As shown, after the host computer sends a control instruction to the USB camera to obtain the RGB image and the depth image, the RGB image (a) and the depth image (b) in the image data are obtained and displayed by analyzing the image data returned by the USB camera.

[0096] According to (a) and (b), the two code streams (RGB image and depth image) can be displayed separately in the interface of the host computer, and the frame rate and field of view of the two code streams are consistent.

[0097] The image display method provided in the embodiment of the present application is applied to a USB camera, wherein the USB camera is equipped with an ITOF module, and the USB camera is used to collect RGB images at a first frame rate and collect depth images at a second frame rate, wherein the first frame rate is greater than or equal to the second frame rate. The method comprises: performing a pairing task at a frequency corresponding to the first frame rate, obtaining a current image pair, wherein the current image pair comprises a current frame RGB image and a target depth image matching the current frame RGB image. According to a pre-calibrated cropping parameter, the target image in the current image pair is cropped, and the processed current image pair is obtained so that the field of view of the two images in the processed current image pair is the same. If the host computer requests to obtain the RGB image and the depth image, the processed current image pair is sent to the host computer according to the preset frame structure, so that the host computer displays the processed current image pair, and the preset frame structure comprises a first frame header and a first frame tail corresponding to the RGB image, and a second frame header and a second frame tail corresponding to the depth image.

[0098] In the present technical solution, by executing the pairing task at a frequency corresponding to the first frame rate, the output frame rates of the RGB image and the depth image can be adjusted to be consistent, so as to ensure that the two can maintain time consistency when displayed on the host computer. Furthermore, the target image in the current image pair is cropped based on the cropping parameters so that the field of view of the two images in the processed current image pair is the same; finally, the two images in the processed current image pair are transmitted to the host computer at the same time through a preset frame structure for display. By adjusting the output frame rate and field of view of the RGB image and the depth image, this solution ensures the effect of synchronous display of the RGB image and the depth image in the host computer, saving equipment costs.

[0099] Optionally, in some embodiments, if the host computer requests to obtain a depth image, the data between the first frame header and the first frame tail is set to a null value, and the depth image in the processed current image pair is placed between the second frame header and the second frame tail to generate image data and send it to the host computer.

[0100] Optionally, the image data can be represented by the following Table 2:

[0101] Table 2

[0102]

[0103] If the host computer requests to obtain an RGB image, the processed RGB image in the current image pair is placed between the first frame header and the first frame tail, and the data between the second frame header and the second frame tail is set to a null value, image data is generated, and sent to the host computer.

[0104] Optionally, the image data can be represented by the following Table 3:

[0105] Table 3

[0106]

[0107] In this embodiment, the operator can send instructions to the USB camera to obtain RGB images and / or depth images according to actual conditions and personal needs, thereby improving the flexibility of the host computer in displaying images.

[0108] Next, the calibration process of the USB camera is explained in detail.

[0109] Figure 7 Schematic diagram of the process of the image display method provided in the embodiment of the present application Figure 2 .like Figure 7 As shown, before S21, the image display method further includes the following steps:

[0110] S71. When calibrating the USB camera according to the calibration mark, determine the image type with the largest field of view angle among the collected calibration RGB image and the calibration depth image as the target image type.

[0111] Exemplarily, the calibration mark may be two opposite vertices of a rectangle, such as an upper left vertex and a lower right vertex.

[0112] For example, the relationship between the field of view angle of the calibrated RGB image and the field of view angle of the calibrated depth image can be referred to Figure 3 It means that it will not be repeated here.

[0113] S72: Determine a size adjustment coefficient according to the target image type, a first coordinate of the calibration mark in the calibration RGB image, and a second coordinate of the calibration mark in the calibration depth image.

[0114] When the target image type is RGB type, S72 can be implemented by following steps 1 to 3:

[0115] Step 1, determining the ratio between the r coordinate difference of the first coordinate and the t coordinate difference of the second coordinate as a first ratio;

[0116] Step 2, determining the ratio between the g coordinate difference of the first coordinate and the f coordinate difference of the second coordinate as the second ratio;

[0117] Step 3: Determine the minimum value of the first ratio and the second ratio as the size adjustment coefficient.

[0118] For example, assuming that the calibration mark is two opposite vertices of a rectangle, such as the upper left vertex and the lower right vertex, the first coordinate of the upper left vertex in the calibration RGB image is (r1", g1"), the first coordinate of the lower right vertex in the calibration RGB image is (r4", g4"), the first coordinate of the upper left vertex in the calibration depth image is (t1", f1"), and the first coordinate of the lower right vertex in the calibration depth image is (t4", f4"). Then the size adjustment coefficient This can be achieved through the following formula:

[0119] =min{(r4"-r1") / (t4"-t1"),(g4"-g1") / (f4"-f1")}

[0120] Among them, (r4"-r1") is the r-coordinate difference of the first coordinate, (t4"-t1") is the t-coordinate difference of the second coordinate, (g4"-g1") is the g-coordinate difference of the first coordinate, and (f4"-f1") is the f-coordinate difference of the second coordinate.

[0121] When the target image type is a depth type, S72 can be implemented by following steps 4 to 7:

[0122] Step 4, determining the ratio between the t-coordinate difference of the second coordinate and the r-coordinate difference of the first coordinate as the third ratio;

[0123] Step 5, determining the ratio between the f-coordinate difference of the second coordinate and the g-coordinate difference of the first coordinate as a fourth ratio;

[0124] Step 6: Determine the maximum value of the third ratio and the fourth ratio as the initial size adjustment coefficient;

[0125] Step 7: The reciprocal of the initial size adjustment coefficient is determined as the size adjustment coefficient.

[0126] Continuing with the above example, the resizing factor This can be achieved through the following formula:

[0127] =1 / max{(t4"-t1") / (r4"-r1"),(f4"-f1") / (g4"-g1")}

[0128] In the above embodiment, considering that the size of the depth image is usually smaller than that of the RGB image, different methods are used to calculate the resizing coefficient for different target image types. When the target image type is RGB type, the minimum value of the first ratio and the second ratio is determined as the resizing coefficient; when the target image type is depth type, the maximum value of the third ratio and the fourth ratio is determined as the initial resizing coefficient, and the reciprocal of the initial resizing coefficient is determined as the resizing coefficient to ensure that the determined target image size is smaller than the initial image size, so that the image cropped according to the resizing coefficient has the same field of view as another uncropped image.

[0129] S73 . Determine a cropping start coordinate according to the target image type, the size adjustment coefficient, the first coordinate of the calibration mark in the calibration RGB image, and the second coordinate of the calibration mark in the calibration depth image.

[0130] In one possible implementation, a target calibration image is determined according to the target image type, and a target calibration marker is determined from the calibration marker. Then, a first coordinate of the target calibration marker in the calibration RGB image and a second coordinate of the target calibration marker in the calibration depth image are determined. Then, the product of the coordinate of the target calibration marker in the non-target calibration image and the size adjustment coefficient is calculated, and the difference between the coordinate of the target calibration marker in the target calibration image and the product is determined as the cropping start coordinate.

[0131] For example, assume that the target image type is RGB type, the target calibration image is the calibration RGB image, the non-target calibration image is the calibration depth image, the target calibration mark is the upper left vertex of the rectangle, the coordinates of the target calibration in the calibration RGB image are (r1", g1"), and the coordinates of the target calibration in the calibration depth image are (t1", f1"). When calculating the cropping start coordinates, first calculate the coordinates of the target calibration mark in the non-target calibration image ((t1", f1")) and the size adjustment coefficient The product of t1", f1")), the target calibration mark is located at the coordinates of the target calibration image ((r1", g1")) and the product (( t1", f1"))) difference ((r1"- t1", g1"- f1")), which is determined as the starting coordinates for clipping.

[0132] It should be understood that reference can be made to Figure 4 , Figure 5 The examples in the illustrated embodiment will not be described in detail here.

[0133] In the above embodiment, the USB camera is calibrated by the calibration mark to obtain the cropping parameters, so that when the RGB image and the depth image are subsequently acquired, the RGB image or the depth image can be cropped based on the cropping parameters to make the field of view of the two the same.

[0134] If the initial calibration marks are the four vertices of a rectangle, in an ideal scenario, the initial calibration marks should form a rectangle in both the calibration RGB image and the calibration depth image. Taking the calibration RGB image as an example, in an ideal scenario, the calibration RGB image should be as follows: Figure 8 shown. Figure 8 This is a schematic diagram of a calibrated RGB image in an ideal scenario provided by an embodiment of the present application. Figure 8 As shown, in an ideal scenario, the figure formed by the initial calibration mark in the calibration RGB image should be a rectangle.

[0135] However, in actual scenes, due to the influence of lens distortion, structural error, etc., the initial calibration mark is located at the position of the calibration RGB image. Fig. 9 shown. Fig. 9 This is a schematic diagram of a calibrated RGB image in an actual scenario provided by an embodiment of the present application. Fig. 9 As shown in the figure, in the actual scene, the initial calibration marks cannot form a rectangle in the calibration RGB image. Therefore, before the USB camera is actually used, its coordinates need to be corrected.

[0136] Fig.10 Schematic diagram of the process of the image display method provided in the embodiment of the present application Figure 3 .like Fig.10 As shown, before S21, the image display method further includes the following steps:

[0137] S101, when calibrating the USB camera according to the initial calibration mark, determining the initial coordinates of the initial calibration mark in the image to be corrected.

[0138] The initial calibration marks are the four vertices of a rectangle, and the image to be corrected is a calibrated depth image or a calibrated RGB image.

[0139] Take the image to be corrected as the calibrated RGB image as an example, refer to Fig. 9 , the initial calibration marks are: (r1,g1), (r2,g2), (r3,g3), (r4,g4).

[0140] S102, calculating the center position between two adjacent vertices of the rectangle according to the initial coordinates of the initial calibration mark in the image to be corrected.

[0141] For example, the center position between two adjacent vertices may be calculated by referring to the following formula:

[0142] r1 =abs(r2-r1) / 2+min(r1,r2)

[0143] g1 =abs(g2-g1) / 2+min(g1,g2)

[0144] r2 =abs(r3-r1) / 2+min(r1,r3)

[0145] g2 =abs(g3-g1) / 2+min(g1,g3)

[0146] r3 =abs(r4-r2) / 2+min(r2,r4)

[0147] g3 =abs(g4-g2) / 2+min(g2,g4)

[0148] r4 =abs(r4-r3) / 2+min(r3,r4)

[0149] g4 =abs(g4-g3) / 2+min(g3,g4)

[0150] Among them, (r1 , g1 ) is the center position between the upper left vertex and the upper right vertex, (r2 , g2 ) is the center position between the upper left vertex and the lower left vertex, (r3 , g3 ) is the center position between the upper right vertex and the lower right vertex, (r4 , g4 ) is the center position between the lower left vertex and the lower right vertex.

[0151] S103. Reconstruct a new rectangle based on the center position.

[0152] In a possible implementation, the center position is determined as the center position of the side length of the new rectangle to construct the new rectangle.

[0153] Fig.11 A schematic diagram of a new rectangle provided in an embodiment of the present application. Fig.11 As shown, Fig.11 The dashed rectangle is the new rectangle.

[0154] S104: Determine two opposite vertices of the new rectangle as calibration marks.

[0155] Optionally, the coordinates of the calibration mark may be determined based on the above-mentioned center position.

[0156] Assuming that the calibration marks are the upper left vertex and the lower right vertex, the coordinates of the calibration marks can be determined by the following formula:

[0157] r1 =r2

[0158] g1 =g1

[0159] r4 =r3

[0160] g4 =g4

[0161] Among them, (r1 ) is the coordinate of the upper left vertex, (r4 ) are the coordinates of the lower right vertex.

[0162] Fig.12 Schematic diagram of a corrected calibrated RGB image provided in an embodiment of the present application. Fig.12 As shown, the upper left vertex and the lower right vertex of the dotted rectangle in the calibration RGB image are calibration marks.

[0163] It should be understood that the method of performing coordinate correction on the calibrated depth image is the same as the method of performing coordinate correction on the calibrated RGB image, which will not be repeated here.

[0164] In the above embodiment, coordinate correction is performed on the calibrated RGB image and the calibrated depth image to ensure the authenticity of the image captured by the USB camera and avoid image distortion.

[0165] Next, the implementation of the image display method is explained in detail through several specific examples.

[0166] First, the image calibration process of the USB camera is explained.

[0167] In practical applications, before calibrating the USB camera, a test environment needs to be built. Fig.13 A schematic diagram of the test environment provided in the embodiment of the present application. Fig.13As shown in the figure, the test environment requires that there are obvious key point identification marks (i.e., initial calibration marks) at the four corners of the test target surface. The distance between the test target surface and the test equipment can be set according to the actual situation. It is necessary to ensure that the test target surface can be displayed completely in both the RGB image and the depth image.

[0168] Fig.14 Schematic diagram of the process of the image display method provided in the embodiment of the present application Figure 4 .like Fig.14 As shown, in the calibration stage, the image display method includes the following steps:

[0169] S141, the host computer sends a start matching process command to the USB camera, and continuously obtains whether the matching process is completed.

[0170] Correspondingly, the USB camera obtains the start matching processing command sent by the host computer.

[0171] S142: The USB camera obtains a frame of calibrated RGB image and a frame of calibrated depth image according to the start matching processing command.

[0172] S143, the USB camera identifies the coordinates of the initial calibration mark in the calibration RGB image and the coordinates of the initial calibration mark in the calibration depth image.

[0173] S144, the USB camera performs coordinate correction on the calibration RGB image and the calibration depth image according to the coordinates of the initial calibration mark in the calibration RGB image and the coordinates of the initial calibration mark in the calibration depth image, and determines the calibration mark.

[0174] S145, the USB camera calculates cropping parameters.

[0175] S146. The USB camera stores the cropping parameters in a flash memory (English: FLASH).

[0176] S147, the USB camera sends a matching process success flag to the host computer.

[0177] The matching process success flag indicates whether the matching process is completed.

[0178] Correspondingly, the host computer obtains a matching processing success flag.

[0179] It should be understood that Fig.14 Each step in the illustrated embodiment has been explained in the above embodiment and will not be described again here.

[0180] Fig.15 Schematic diagram of the process of the image display method provided in the embodiment of the present application Figure 5 .like Fig.15As shown, in the actual use stage, the image display method includes the following steps:

[0181] S151, power on the USB camera.

[0182] S152: The host computer sends a control command to the USB camera.

[0183] The control instructions include "display only RGB image", "display only depth image", and "display both RGB image and depth image".

[0184] Correspondingly, the USB camera obtains the control instructions sent by the host computer.

[0185] S153, the USB camera collects and executes the pairing task based on the control instruction to obtain the current image pair.

[0186] S154: The USB camera obtains cropping parameters from the FLASH, and crops the target image in the current image pair based on the cropping parameters to generate a processed current image pair.

[0187] S155. The USB camera sends the processed RGB image and / or depth image in the current image pair to the host computer according to the control instruction.

[0188] S156: The host computer receives the RGB image and / or the depth image in the current image pair sent by the USB camera, and displays it.

[0189] It should be understood that Fig.15 Each step in the illustrated embodiment has been explained in the above embodiment and will not be described again here.

[0190] Fig.16 A schematic diagram of the structure of the image display device provided in this application, such as Fig.16 As shown, the image display device 160 provided in this embodiment is set in a USB camera, the USB camera is deployed with an ITOF module, the USB camera is used to collect RGB images according to a first frame rate, and collect depth images according to a second frame rate, the first frame rate is greater than or equal to the second frame rate, and the image display device 160 includes:

[0191] The task execution module 161 is used to execute the pairing task according to the frequency corresponding to the first frame rate, and obtain a current image pair, where the current image pair includes a current frame RGB image and a target depth image matching the current frame RGB image;

[0192] A cropping module 162, used to crop the target image in the current image pair according to a pre-calibrated cropping parameter to obtain a processed current image pair so that the field of view angles of the two images in the processed current image pair are the same;

[0193] A sending module 163 is used for sending the processed current image pair to the host computer according to a preset frame structure if the host computer requests to obtain the RGB image and the depth image, so that the host computer displays the processed current image pair. The preset frame structure includes a first frame header and a first frame footer corresponding to the RGB image, and a second frame header and a second frame footer corresponding to the depth image;

[0194] Among them, the pairing task is: add 1 to i as the new i, if the new i is less than the preset value, the current frame depth image is determined as the target depth image; if the new i is greater than or equal to the preset value, the previous frame depth image is determined as the target depth image, and the new i is reset to zero; the preset value is the ratio of the first frame rate to the preset frequency difference, the preset frequency difference is the difference between the first frame rate and the second frame rate, and the preset value is greater than 1.

[0195] In a possible implementation, the cropping parameters include a target image type, a resizing coefficient, and a cropping start coordinate, and the target image type includes a depth type or an RGB type; the cropping module 162 is specifically configured to:

[0196] Determine the target image in the current image pair according to the target image type;

[0197] The target image is cropped according to the cropping start coordinates to crop the target image to a target image size, and a processed current image pair is obtained, where the target image size is the product of an initial image size of a non-target image in the current image pair and a size adjustment coefficient.

[0198] In a possible implementation, before performing the pairing task at a frequency corresponding to the first frame rate to obtain the current image pair, the image display device 160 further includes a calibration module for:

[0199] When the USB camera is calibrated according to the calibration mark, the image type with the largest field of view angle among the collected calibration RGB image and calibration depth image is determined as the target image type;

[0200] Determining a resizing factor based on the target image type, a first coordinate of the calibration marker in the calibration RGB image, and a second coordinate of the calibration marker in the calibration depth image;

[0201] The cropping start coordinates are determined according to the target image type, the size adjustment coefficient, the first coordinates of the calibration mark in the calibration RGB image, and the second coordinates of the calibration mark in the calibration depth image.

[0202] In a possible implementation manner, the calibration mark is two opposite vertices of a rectangle, and the calibration module is specifically used to:

[0203] When the target image type is an RGB type, a ratio between an r coordinate difference value of the first coordinate and a t coordinate difference value of the second coordinate is determined as a first ratio;

[0204] Determine the ratio between the g-coordinate difference of the first coordinate and the f-coordinate difference of the second coordinate as a second ratio;

[0205] determining the minimum value of the first ratio and the second ratio as a size adjustment coefficient;

[0206] When the target image type is a depth type, determining a ratio between a t-coordinate difference value of the second coordinate and an r-coordinate difference value of the first coordinate as a third ratio;

[0207] determining a ratio between the f-coordinate difference value of the second coordinate and the g-coordinate difference value of the first coordinate as a fourth ratio;

[0208] determining the maximum value of the third ratio and the fourth ratio as the initial size adjustment coefficient;

[0209] The reciprocal of the initial size adjustment factor is determined as the size adjustment factor.

[0210] In a possible implementation, before performing the pairing task at a frequency corresponding to the first frame rate to obtain the current image pair, the calibration module is further used to:

[0211] When calibrating the USB camera according to the initial calibration mark, determining the initial coordinates of the initial calibration mark in the image to be corrected, the initial calibration mark is four vertices of a rectangle, and the image to be corrected is a calibrated depth image or a calibrated RGB image;

[0212] Calculate the center position between two adjacent vertices of the rectangle according to the initial coordinates of the initial calibration mark in the image to be corrected;

[0213] Reconstruct a new rectangle based on the center position;

[0214] Two opposite vertices of the new rectangle are determined as calibration marks.

[0215] In a possible implementation manner, the sending module 163 is specifically configured to:

[0216] If the host computer requests to obtain the RGB image and the depth image, the RGB image in the processed current image pair is placed between the first frame header and the first frame tail, and the depth image in the processed current image pair is placed between the second frame header and the second frame tail, the image data is generated, and sent to the host computer.

[0217] In a possible implementation manner, the sending module 163 is further configured to:

[0218] If the host computer requests to obtain a depth image, the data between the first frame header and the first frame tail is set to a null value, and the processed depth image in the current image pair is placed between the second frame header and the second frame tail, image data is generated, and sent to the host computer;

[0219] If the host computer requests to obtain an RGB image, the processed RGB image in the current image pair is placed between the first frame header and the first frame tail, and the data between the second frame header and the second frame tail is set to a null value, image data is generated, and sent to the host computer.

[0220] The image display device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be described in detail here.

[0221] Fig.17 This is a schematic diagram of the structure of the USB camera provided by this application. Fig.17 As shown, the USB camera 170 provided in this embodiment includes: at least one processor 171, a memory 172 and an ITOF module 173. Optionally, the device 17 also includes a communication component 174. The processor 171, the memory 172 and the communication component 174 are connected via a bus 175.

[0222] Among them, the ITOF module 173 is used to collect depth images.

[0223] In a specific implementation process, at least one processor 171 executes the computer-executable instructions stored in the memory 172, so that at least one processor 171 executes the above method.

[0224] The specific implementation process of the processor 171 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0225] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0226] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0227] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0228] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0229] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0230] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0231] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0232] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0233] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0234] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0235] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0236] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0237] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image display method, characterized in that: Applied to a universal serial bus (USB) camera, the USB camera is equipped with an indirect time-of-flight (ITOF) module, the USB camera is used to collect red, green, and blue (RGB) images at a first frame rate, and collect depth images at a second frame rate, the first frame rate is greater than or equal to the second frame rate, the method comprises: Perform a pairing task at a frequency corresponding to the first frame rate to obtain a current image pair, where the current image pair includes a current frame RGB image and a target depth image matching the current frame RGB image; Cropping the target image in the current image pair according to pre-calibrated cropping parameters to obtain a processed current image pair, so that the field of view angles of the two images in the processed current image pair are the same; If the host computer requests to obtain the RGB image and the depth image, the processed current image pair is sent to the host computer according to the preset frame structure, so that the host computer displays the processed current image pair, and the preset frame structure includes a first frame header and a first frame footer corresponding to the RGB image, and a second frame header and a second frame footer corresponding to the depth image; Among them, the pairing task is: add 1 to i as the new i, if the new i is less than a preset value, determine the current frame depth image as the target depth image; if the new i is greater than or equal to the preset value, determine the previous frame depth image as the target depth image, and reset the new i to zero; the preset value is the ratio of the first frame rate to the preset frequency difference, the preset frequency difference is the difference between the first frame rate and the second frame rate, and the preset value is greater than 1.

2. The method according to claim 1, characterized in that The cropping parameters include a target image type, a size adjustment coefficient, and a cropping start coordinate, and the target image type includes a depth type or an RGB type; and the step of cropping the target image in the current image pair according to the pre-calibrated cropping parameters to obtain the processed current image pair includes: Determining the target image in the current image pair according to the target image type; The target image is cropped according to the cropping start coordinates to crop the target image to a target image size, and the processed current image pair is obtained, where the target image size is the product of an initial image size of a non-target image in the current image pair and the size adjustment coefficient.

3. The method according to claim 2, characterized in that Before performing the pairing task at the frequency corresponding to the first frame rate to obtain the current image pair, the method further includes: When the USB camera is calibrated according to the calibration mark, the image type with the largest field of view angle in the collected calibration RGB image and calibration depth image is determined as the target image type; Determining the resizing factor according to the target image type, a first coordinate of the calibration mark in the calibration RGB image, and a second coordinate of the calibration mark in the calibration depth image; The cropping start coordinates are determined according to the target image type, the size adjustment coefficient, the first coordinates of the calibration mark in the calibration RGB image, and the second coordinates of the calibration mark in the calibration depth image.

4. The method according to claim 3, characterized in that: The calibration mark is two opposite vertices of a rectangle, and determining the size adjustment coefficient according to the target image type, the first coordinate of the calibration mark in the RGB image, and the second coordinate of the calibration mark in the depth image includes: When the target image type is the RGB type, determining a ratio between an r coordinate difference value of the first coordinate and a t coordinate difference value of the second coordinate as a first ratio; Determine the ratio between the g-coordinate difference value of the first coordinate and the f-coordinate difference value of the second coordinate as a second ratio; Determine the minimum value of the first ratio and the second ratio as the size adjustment coefficient; When the target image type is the depth type, determining a ratio between a t-coordinate difference value of the second coordinate and an r-coordinate difference value of the first coordinate as a third ratio; determining a ratio between the f-coordinate difference value of the second coordinate and the g-coordinate difference value of the first coordinate as a fourth ratio; Determine the maximum value of the third ratio and the fourth ratio as an initial size adjustment coefficient; The reciprocal of the initial size adjustment coefficient is determined as the size adjustment coefficient.

5. The method according to any one of claims 1 to 4, characterized in that: Before performing the pairing task at the frequency corresponding to the first frame rate to obtain the current image pair, the method further includes: When calibrating the USB camera according to the initial calibration mark, determining the initial coordinates of the initial calibration mark in the image to be corrected, wherein the initial calibration mark is four vertices of a rectangle, and the image to be corrected is a calibrated depth image or a calibrated RGB image; Calculating the center position between two adjacent vertices of the rectangle according to the initial coordinates of the initial calibration mark in the image to be corrected; Based on the center position, reconstruct a new rectangle; Two opposite vertices of the new rectangle are determined as the calibration marks.

6. The method according to any one of claims 1 to 4, characterized in that: If the host computer requests to obtain the RGB image and the depth image, the processed current image pair is sent to the host computer according to the preset frame structure, including: If the host computer requests to obtain the RGB image and the depth image, the RGB image in the processed current image pair is placed between the first frame header and the first frame tail, and the depth image in the processed current image pair is placed between the second frame header and the second frame tail, image data is generated, and sent to the host computer.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: If the host computer requests to obtain a depth image, the data between the first frame header and the first frame tail is set to a null value, and the depth image in the processed current image pair is placed between the second frame header and the second frame tail, image data is generated, and sent to the host computer; If the host computer requests to obtain an RGB image, the RGB image in the processed current image pair is placed between the first frame header and the first frame tail, and the data between the second frame header and the second frame tail is set to a null value, image data is generated, and sent to the host computer.

8. An image display device, characterized in that: Applied to a universal serial bus (USB) camera, the USB camera is equipped with an indirect time-of-flight (ITOF) module, the USB camera is used to collect red, green, and blue (RGB) images at a first frame rate, and collect depth images at a second frame rate, the first frame rate is greater than or equal to the second frame rate, and the device comprises: A task execution module, configured to execute a pairing task at a frequency corresponding to the first frame rate to obtain a current image pair, wherein the current image pair includes a current frame RGB image and a target depth image matching the current frame RGB image; A cropping module, used for cropping the target image in the current image pair according to a pre-calibrated cropping parameter to obtain a processed current image pair so that the field of view angles of the two images in the processed current image pair are the same; A sending module, for sending the processed current image pair to the host computer according to a preset frame structure if the host computer requests to obtain the RGB image and the depth image, so that the host computer displays the processed current image pair, wherein the preset frame structure includes a first frame header and a first frame footer corresponding to the RGB image, and a second frame header and a second frame footer corresponding to the depth image; Among them, the pairing task is: add 1 to i as the new i, if the new i is less than a preset value, determine the current frame depth image as the target depth image; if the new i is greater than or equal to the preset value, determine the previous frame depth image as the target depth image, and reset the new i to zero; the preset value is the ratio of the first frame rate to the preset frequency difference, the preset frequency difference is the difference between the first frame rate and the second frame rate, and the preset value is greater than 1.

9. A USB camera, characterized in that: include: Memory, processor and indirect time-of-flight ITOF module; The ITOF module is used to collect depth images; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images