Image capture system, method, computing device, and storage medium

Abstract

The application provides an image acquisition system, method, computing device and storage medium. The system comprises: a first image acquisition device configured to acquire at least two frames of depth images of a target object; a plurality of second image acquisition devices configured to acquire images of the target object from different angles; and a controller configured to determine a displacement of the target object based on the at least two frames of depth images, and control the plurality of second image acquisition devices to deflect respectively following the target object based on the displacement, so that the target object is at least within a common field of view range of the plurality of second image acquisition devices, and control the plurality of second image acquisition devices to acquire images of the target object from different angles. The first image acquisition device is used to obtain the displacement of the target object, and the plurality of second image acquisition devices are controlled to deflect following the target object, so that the target object can have a larger range of activities, and a better user experience is provided.

Description

Technical Field

[0001] This invention relates to the field of holographic imaging technology, and in particular to an image acquisition system, method, computing device, and storage medium. Background Technology

[0002] 3D displays are immersive displays for the future and are currently under development. Related technologies typically involve fixing 4-6 RGB or RGBD cameras around the bezel of the 3D display to collect image data of a person from different perspectives. The collected image data is then encoded and sent to a neural network model for 3D reconstruction of the person.

[0003] However, the fixed installation positions and angles of these cameras restrict users to a very small area around the intersection of their optical axes. Excessive movement can easily push them out of the field of view of any camera, resulting in missing image data. Furthermore, since current neural network models are trained using image data from fixed-position cameras, the camera positions and angles must remain constant during use. Users must move only within this narrow area for the captured images to be used for 3D reconstruction by the neural network model; otherwise, reconstruction will fail, preventing the user from seeing a complete 3D image. This forces users to maintain a single posture for extended periods, impacting user experience and hindering the widespread adoption of the technology.

[0004] Therefore, a new image acquisition system is needed to at least solve the above problems. Summary of the Invention

[0005] The main objective of this invention is to provide an image acquisition system, method, computing device, and storage medium to increase the range of motion of the image acquisition object.

[0006] The present invention provides an image acquisition system, comprising: a first image acquisition device for acquiring at least two frames of depth images of a target object; a plurality of second image acquisition devices for acquiring images of the target object from multiple different angles; and a controller for determining the displacement of the target object based on the at least two frames of depth images, and controlling the plurality of second image acquisition devices to deflect according to the displacement to follow the target object so that the target object is at least within the common field of view of the plurality of second image acquisition devices, and controlling the plurality of second image acquisition devices to acquire images of the target object from multiple different angles.

[0007] In one embodiment, the horizontal field of view of the first image acquisition device is not less than 120°, and / or its vertical field of view is not less than 90°.

[0008] In one embodiment, the device further includes: a first gimbal, which is mechanically connected to a first image acquisition device to drive the first image acquisition device to deflect; the controller is also configured to control the movement of the first gimbal according to the displacement so that the first gimbal drives the first image acquisition device to deflect following the target object.

[0009] In one embodiment, the system further includes: a plurality of second gimbals, each of which is mechanically connected to a plurality of second image acquisition devices to drive the plurality of second image acquisition devices to deflect respectively; the controller is further configured to control the movement of the plurality of second gimbals according to the displacement of the target object, so that the plurality of second gimbals drive the plurality of second image acquisition devices to deflect respectively following the target object.

[0010] In one embodiment, each second gimbal includes at least three rotational degrees of freedom.

[0011] In one embodiment, the controller is further configured to: acquire the pixel coordinates and depth value of the target object for each frame of depth image; determine the three-dimensional coordinates of the target object in the three-dimensional coordinate system of the first image acquisition device based on the pixel coordinates and depth value of the target object; and determine the displacement of the target object based on the three-dimensional coordinates of the target object corresponding to at least two frames of depth images.

[0012] In one embodiment, the target object includes a target human body; the controller is further configured to sequentially perform grayscale processing and binarization processing on each frame of depth image to obtain at least two binarized images, and extract the contour images of the target human body in the at least two binarized images respectively, select a reference point in the contour image, and use the position change of the reference point in the at least two frames of depth images as the displacement of the target object.

[0013] In one embodiment, the reference point includes the center point of the neck of the target human body.

[0014] In one embodiment, the controller is further configured to determine the deflection angle of each of the plurality of second image acquisition devices based on the displacement, and control the plurality of second image acquisition devices to deflect according to their respective deflection angles.

[0015] In one embodiment, the controller is further configured to, for each second image acquisition device, determine the displacement projection in the three-dimensional coordinate system of the second image acquisition device based on the pre-calibrated extrinsic parameters of the second image acquisition device relative to the three-dimensional coordinate system of the first image acquisition device; and determine the deflection angle of the second image acquisition device based on the normal vector of the three-dimensional coordinate system of the second image acquisition device and the displacement projection.

[0016] In one embodiment, the controller is further configured to determine the deflection angle of the second image acquisition device based on the fact that the displacement of the normal vector of the three-dimensional coordinate system of the second image acquisition device along the displacement projection is equivalent to the rotation of the normal vector around each coordinate axis.

[0017] In one embodiment, the controller is further configured to update the extrinsic parameters of the plurality of second image acquisition devices relative to the first image acquisition device based on the deflection angle of each of the plurality of second image acquisition devices.

[0018] In one embodiment, the controller is further configured to compare the displacement distance of the target object with a preset distance threshold, and if the displacement distance of the target object exceeds the preset distance threshold, control multiple second image acquisition devices to follow the target object and deflect accordingly.

[0019] In one embodiment, the controller is also configured to generate a three-dimensional image of the target object using images of the target object from multiple different angles.

[0020] The present invention provides an image acquisition method based on the above-mentioned image acquisition system, comprising: acquiring at least two depth images of a target object; determining the displacement of the target object based on the at least two depth images; controlling a plurality of second image acquisition devices to follow the target object and deflect accordingly, so that the target object is at least within the common field of view of the plurality of second image acquisition devices; and controlling the plurality of second image acquisition devices to acquire images of the target object from multiple different angles.

[0021] The present invention provides a computing device, characterized in that it includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the above-described image acquisition method is implemented.

[0022] The present invention provides a storage medium storing a computer program, which, when executed by a processor, implements the above-described image acquisition method.

[0023] Using the image acquisition system of this embodiment, the displacement of the target object can be obtained by the first image acquisition device, and multiple second image acquisition devices can be controlled to follow the deflection of the target object, so that the target object can have a larger range of movement and bring a better user experience. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0025] Figure 1 This is a schematic diagram of the arrangement of an image acquisition system according to an embodiment of this application;

[0026] Figure 2 To utilize Figure 1A schematic diagram comparing the range of motion of the target object obtained by the image acquisition system shown with the range of motion of the target object in related technologies;

[0027] Figure 3 This is a schematic diagram of a gimbal according to a specific embodiment of this application;

[0028] Figure 4 This is a schematic diagram illustrating the determination of the three-dimensional coordinates of a target object in the three-dimensional coordinate system of a first image acquisition device according to an embodiment of this application.

[0029] Figure 5 This is a schematic diagram of a method for determining the displacement of a target object according to an embodiment of this application;

[0030] Figures 6A to 6C This is a schematic diagram of a method for determining a reference point according to an embodiment of this application;

[0031] Figure 7 This is a schematic diagram of a neural network model for 3D reconstruction using images acquired from different angles, according to an embodiment of this application.

[0032] Figure 8 This is a schematic diagram of a chessboard grid according to one embodiment of this application;

[0033] Figure 9 This is a flowchart of an image acquisition method according to an embodiment of this application. Detailed Implementation

[0034] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0035] This embodiment provides an image acquisition system, including: a first image acquisition device for acquiring at least two frames of depth images of a target object; a plurality of second image acquisition devices for acquiring images of the target object from multiple different angles; and a controller for determining the displacement of the target object based on the at least two frames of depth images, and controlling the plurality of second image acquisition devices to follow the target object and deflect accordingly, so that the target object is at least within the common field of view of the plurality of second image acquisition devices, and controlling the plurality of second image acquisition devices to acquire images of the target object from multiple different angles.

[0036] In this embodiment, the first image acquisition device may include a depth camera, such as an infrared camera or infrared camera. The second image acquisition device may include an RGB or RGBD camera, RGB or RGBD camera, etc. The actual devices used for the first and second image acquisition devices in this embodiment are not specifically limited, as long as they can achieve the functions required by this embodiment, they are within the scope of protection of this application.

[0037] In this embodiment, the target object can be a human body, a small animal, etc., and this application does not specifically limit it. The first image acquisition device can acquire at least two frames (e.g., one frame every five frames) of depth images of the target object at a preset frequency (e.g., 60Hz), or it can acquire at least two frames of depth images of the target object at preset time intervals, or it can randomly acquire at least two frames of depth images of the target object. Those skilled in the art can set it as needed, and this application does not specifically limit it.

[0038] In this embodiment, multiple second image acquisition devices acquire images of the target object from multiple different angles. Those skilled in the art can set the angle of the acquired images as needed, that is, those skilled in the art can set the positional distribution of the multiple second image acquisition devices as needed. This application does not make specific limitations in this regard.

[0039] refer to Figure 1 The first image acquisition device C0 can be set at the center and above the screen S, and its optical axis can be horizontal (i.e. perpendicular to the screen S) to acquire at least two frames of depth images of the target object in front of the screen S; multiple second image acquisition devices C1 to C4 can be set at the four corners of the screen S to acquire images of the target object in front of the screen S from different angles.

[0040] Compared to using a fixed camera to capture images, in this embodiment, the displacement of the target object is determined by a first image acquisition device, thereby determining the reasonable location of the target object. By controlling multiple second image acquisition devices to follow the target object's deflection based on its displacement, the multiple second image acquisition devices have a larger common field of view. That is, the multiple second image acquisition devices can jointly acquire a larger field of view of the target object's image. Correspondingly, the target object's range of motion is also larger, and it is easier to achieve a complete reconstruction of the target object's 3D image, which can bring a better user experience to the user of this image acquisition system.

[0041] refer to Figure 2In related technologies, each camera is fixed and the common field of view is only the area shown in OR. According to the movable range of the optical axis OA1 of the first image acquisition device and the movable range of the optical axis OA2 of the multiple second image acquisition devices in this embodiment, each second image acquisition device can follow the target object and make corresponding deflections. The common field of view then becomes the area shown in NR. Obviously, the area shown in NR is larger than the area shown in OR, that is, the target object can have a larger range of movement.

[0042] In one embodiment, the horizontal field of view of the first image acquisition device may be no less than 120°, and / or its vertical field of view may be no less than 90°. In other embodiments, the selected first image acquisition device may also have a larger or smaller horizontal or vertical field of view, for example, the horizontal and vertical field of view of the first image acquisition device may both be 120°, 150°, etc., and this application does not specifically limit this.

[0043] By selecting the horizontal and vertical field of view of the first image acquisition device, the movable range of the target object can be affected. If a larger movable range of the target object is desired, a larger horizontal and vertical field of view can be selected; if a larger movable range of the target object is not required, a smaller horizontal and vertical field of view can be selected. This application does not impose specific limitations in this regard. By selecting appropriate horizontal and vertical field of view of the first image acquisition device, the movable range of the target object can be appropriately controlled.

[0044] In one embodiment, the image acquisition system may further include: a first pan-tilt unit, mechanically connected to a first image acquisition device to drive the first image acquisition device to deflect; the controller is further configured to control the movement of the first pan-tilt unit according to displacement, so that the first pan-tilt unit drives the first image acquisition device to follow the target object in deflection. In one embodiment, the first pan-tilt unit may include at least three degrees of freedom, which may include rotation, translation, etc., and this application does not specifically limit them.

[0045] By setting a first pan-tilt unit for the first image acquisition device, the field of view of the first image acquisition device can be further increased on the basis of its original field of view, thereby increasing the range of motion of the target object.

[0046] In one embodiment, the image acquisition system may further include: a plurality of second pan-tilt units, each mechanically connected to a plurality of second image acquisition devices in a one-to-one correspondence, to drive the plurality of second image acquisition devices to deflect respectively; the controller is further configured to control the movement of the plurality of second pan-tilt units according to the displacement of the target object, so that the plurality of second pan-tilt units drive the plurality of second image acquisition devices to deflect respectively following the target object. In one embodiment, each second pan-tilt unit includes at least three rotational degrees of freedom. In other embodiments, each second pan-tilt unit may also include more or fewer degrees of freedom, and may also include translational degrees of freedom, etc.

[0047] refer to Figure 3 The styles of the first and second gimbals can be as follows: Figure 3 As shown.

[0048] In one embodiment, the controller can also be used to acquire the pixel coordinates and depth value of the target object for each frame of depth image, determine the three-dimensional coordinates of the target object in the three-dimensional coordinate system of the first image acquisition device based on the pixel coordinates and depth value of the target object, and determine the displacement of the target object based on the three-dimensional coordinates of the target object corresponding to at least two frames of depth images.

[0049] refer to Figure 4 For a point A on the depth image Pic, its pixel coordinates are (ua, va), and its depth value is |Z|. a The pixel coordinates of point A can be converted into three-dimensional coordinates in the three-dimensional coordinate system of the first image acquisition device through the following steps:

[0050] (1) First, calculate the projection length of point A on the z-axis of the three-dimensional coordinate system of the first image acquisition device, which is the distance from the origin Oc of the three-dimensional coordinate system to the image center Ouv(u0, v0) |Z c_a |

[0051] When the plane containing the depth image Pic is perpendicular to the z-axis of the three-dimensional coordinate system Oc of the first image acquisition device, according to the Pythagorean theorem, the projection length of point A on the z-axis of the three-dimensional coordinate system of the first image acquisition device can be obtained as follows: in,

[0052] Even if the plane containing the depth image Pic is not perpendicular to the z-axis of the three-dimensional coordinate system Oc of the first image acquisition device, the angle between the plane containing Pic and the xy plane of the three-dimensional coordinate system Oc of the first image acquisition device can still be calculated, and the projection of point A on the z-axis of the three-dimensional coordinate system Oc can also be obtained.

[0053] (2) The three-dimensional coordinates P of point A in the three-dimensional coordinate system of the first image acquisition device are determined by the following formula.A (X c_a Y c_a Z c_a ):

[0054]

[0055] Among them, K -1 Z represents the inverse matrix of the intrinsic parameters of the first image acquisition device. c_a This represents a scaling factor.

[0056] refer to Figure 5 Based on the two three-dimensional coordinates P of point A in the three-dimensional coordinate system of the first image acquisition device CAMERA, corresponding to the two depth images, A -new and P A -old, we can obtain the displacement M of point A = (P A -new)-(P A -old).

[0057] By calculating the displacement of the target object in three-dimensional space, a more accurate positional change of the target object can be obtained, which helps to adjust the focal length of multiple second image acquisition devices, acquire images with higher clarity, and achieve better 3D reconstruction results.

[0058] Of course, in other embodiments, the displacement of the target object can be determined solely based on changes in its pixel coordinates. In this case, the displacement of the target object is a displacement within a two-dimensional plane. Those skilled in the art can choose to calculate the displacement of the target object in three-dimensional space or in two-dimensional space as needed, and this application does not specifically limit this.

[0059] In one embodiment, the controller can also be used to compare the displacement distance of the target object with a preset distance threshold, and if the displacement distance of the target object exceeds the preset distance threshold, control multiple second image acquisition devices to follow the target object and deflect accordingly.

[0060] In this embodiment, multiple second image acquisition devices are controlled to follow the target object and deflect accordingly only when the displacement distance of the target object is determined to be large. This appropriately reduces the amount of calculation and the amount of movement of the multiple second image acquisition devices, thereby improving the service life of the multiple second image acquisition devices.

[0061] In one embodiment, the target object may include a target human body; the controller is further configured to sequentially perform grayscale processing and binarization processing on each frame of depth image to obtain at least two binarized images, and extract the contour images of the target human body in the at least two binarized images respectively, distinguish the target human body from the background, select a reference point in the contour image, and use the position change of the reference point in the at least two frames of depth images as the displacement of the target object.

[0062] refer to Figures 6A to 6C Depth images such as Figure 6A As shown in the image (color image), the image after grayscale conversion is as follows: Figure 6B As shown, the grayscale value gray = (R + G + B) / 3; then, the grayscale image is converted into a binary image by a set binarization threshold, as shown. Figure 6C As shown. When the target object is a human body, the neck of the target human body has relatively less movement compared to other parts (the position of the neck is not easily affected by body twisting). Therefore, it can be used to determine whether to deflect the reference points of multiple second image acquisition devices. Thus, the center point of the human neck can be selected as the reference point. When the center point of the human neck moves, it indicates that the human body has moved significantly, requiring deflection of each of the multiple second image acquisition devices. (Reference) Figure 6C In one embodiment, starting from the top O of the human image, two points A and B can be used to move along the left and right contours from point O, respectively, and the distance between points A and B can be calculated in real time. When the shortest distance between the two points is obtained, it indicates the location of the neck. The coordinates of the neck center can be obtained based on the coordinates of points A and B. If the target object is another object, a reference point can be determined according to the situation, and this application does not make specific limitations in this regard.

[0063] By selecting a reference point, the displacement of the reference point can be used to represent the displacement of the target object, thus making it easier to determine the displacement of the target object.

[0064] In one embodiment, the controller can also be used to determine the deflection angle of each of the plurality of second image acquisition devices based on the displacement, and control the plurality of second image acquisition devices to deflect according to their respective deflection angles.

[0065] In this embodiment, the multiple second image acquisition devices have different relative positional relationships with the target object. Therefore, if the target object is displaced, and the multiple second image acquisition devices are to follow the target object, they can be deflected individually, for example, by rotating around the x-axis, y-axis, and z-axis. The deflection angle of each second image acquisition device can then be determined based on the displacement of the target object, thereby achieving a larger common field of view for the multiple second image acquisition devices, and thus a larger range of motion for the target object.

[0066] In other embodiments, the controller can also be used to determine the deflection angle and displacement of each of the multiple second image acquisition devices based on the displacement of the target object, so that the movement of the multiple second image acquisition devices includes deflection and displacement, thereby realizing a more flexible movement mode of the second image acquisition devices.

[0067] In one embodiment, the controller can also be used to determine, for each second image acquisition device, the displacement projection in the three-dimensional coordinate system of the second image acquisition device based on the pre-calibrated extrinsic parameters of the second image acquisition device relative to the three-dimensional coordinate system of the first image acquisition device; and to determine the deflection angle of the second image acquisition device based on the normal vector of the three-dimensional coordinate system of the second image acquisition device and the displacement projection.

[0068] In this embodiment, the displacement of the target object in the three-dimensional coordinate system of the second image acquisition device is determined by first converting the displacement of the target object into a unit vector using the following formula:

[0069]

[0070] Subsequently, the unit vector M of the target object's displacement can be calculated using the following formula. n Displacement projection in the three-dimensional coordinate system of the second image acquisition device:

[0071] M n1 =M n *E 10

[0072] Among them, E 10 This represents the extrinsic parameter matrix of the second image acquisition device C1 relative to the first image acquisition device C0.

[0073] Finally, the deflection angle of the second image acquisition device is determined based on the normal vector and displacement projection of the three-dimensional coordinate system of the second image acquisition device. In one embodiment, the deflection angle of the second image acquisition device can be determined by the fact that the displacement of the normal vector of the three-dimensional coordinate system of the second image acquisition device along the displacement projection is equivalent to the rotation of the normal vector around each coordinate axis. That is, the deflection angle of the second image acquisition device around each coordinate axis is determined by the fact that the displacement transformation of the normal vector of the three-dimensional coordinate system of the second image acquisition device along the displacement projection is the same as the result obtained by the rotation transformation of the normal vector around each coordinate axis.

[0074] For example, for the second image acquisition device C1, its unit normal vector n0 can be:

[0075] Of course, in other implementations, the elements of the unit normal vector can also be other values; this is only used as an example.

[0076] We can first assume that the rotation matrices around the three coordinate axes are M. x M y M z Let r, p, and y represent the rotation angles around the three coordinate axes, then we have:

[0077]

[0078]

[0079]

[0080] Then, the following formula is used to calculate each rotation matrix and the new normal vector n of the second image acquisition device C. 1_new :

[0081] n 1_new =n1-M n1 =M X *M Y *M Z *n1

[0082] Thus, we obtain three rotation angles r, p, and y.

[0083] The calculations for the other second image acquisition devices C2, C3, and C4 can also be performed using the same method.

[0084] In one embodiment, the controller can also be used to update the extrinsic parameters of the plurality of second image acquisition devices relative to the first image acquisition device based on the deflection angle of each of the plurality of second image acquisition devices.

[0085] In this embodiment, for example, for the second image acquisition device C1, its extrinsic parameters relative to the first image acquisition device can be updated using the following formula:

[0086] E 10_new =E 10 *M x *M y *M z

[0087] Among them, E 10_new This indicates the new external parameters of the second image acquisition device C1 relative to the first image acquisition device.

[0088] In one embodiment, the controller can also be used to generate a three-dimensional image of the target object using images of the target object from multiple different angles.

[0089] In this embodiment, a 3D image reconstruction neural network model is first trained using a 3D image of the target object and images from multiple different angles to obtain a trained 3D image reconstruction neural network model; subsequently, a reference... Figure 7 The newly acquired target object is input into the trained 3D image reconstruction neural network model from multiple different angles to output the reconstructed 3D image. Figure 7 The 3D image reconstruction neural network model in the image includes an input layer, hidden layer 1, hidden layer 2, and an output layer. Figure 7 The neural network structure shown is only an example. In other implementations, more hidden layers, fully connected layers, pooling layers, etc. may be included. Those skilled in the art can set it up as needed, and this application does not make any specific limitations.

[0090] Before using the image acquisition system of this embodiment to acquire images, the internal and external parameters of the first image acquisition device and the multiple second image acquisition devices can be calibrated.

[0091] In one implementation, it can be utilized as follows: Figure 8 The chessboard pattern shown is used for calibration of intrinsic and extrinsic parameters. Multiple sets of images are recorded simultaneously by a person holding the chessboard and moving it in front of a camera, captured by the first and second image acquisition devices. These images are then fed into a MATLAB calibration program. The program stops when the calibration error decreases to a predetermined range. Of course, those skilled in the art can use other methods for calibration, and this application does not impose specific limitations on them.

[0092] Using the image acquisition system of this embodiment, the displacement of the target object can be obtained by the first image acquisition device, and multiple second image acquisition devices can be controlled to follow the deflection of the target object, thereby enabling the multiple second image acquisition devices to have a larger common field of view, allowing the target object to have a larger range of movement and bringing a better user experience.

[0093] This embodiment provides an image acquisition method based on the above-described image acquisition system, including:

[0094] S100: Acquire at least two depth images of the target object.

[0095] S200: Determine the displacement of the target object based on at least two depth images, and control multiple second image acquisition devices to deflect according to the displacement so that the target object is at least within the common field of view of the multiple second image acquisition devices.

[0096] S300: Controls multiple second image acquisition devices to acquire images of the target object from multiple different angles.

[0097] For example, this explanation will be based on the scenario where both the first and second image acquisition devices are cameras. (Reference) Figure 9 The first image acquisition device can be an infrared camera, and the second image acquisition device can be an RGB camera. First, the infrared camera and each RGB camera are calibrated. Then, the infrared camera acquires an image of the target person. The controller processes the image of the target person to determine the displacement distance of the target person. If the displacement distance of the target person is greater than a distance threshold T, the deflection angle of each RGB camera is calculated, and each RGB camera is controlled to deflect accordingly. After deflection, the RGB cameras acquire images of the target person from different angles. Finally, the acquired images are input into a 3D image reconstruction neural network model for 3D image reconstruction, and the reconstructed 3D image is output and displayed.

[0098] In the process of processing the image of the target person, the image can first be processed by grayscale, binarization, and contour recognition of the target person using the Canny operator. Then, the center point of the neck of the target person is determined as a reference point based on the contour of the target person, and the displacement of the target person is calculated based on the coordinates of the center point of the neck.

[0099] This embodiment provides a computing device, including a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, the above-described image acquisition method is implemented.

[0100] In one embodiment, the computing device may include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0101] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash FLASH RAM). Memory is an example of computer-readable media.

[0102] This embodiment provides a storage medium storing a computer program. When the computer program is executed by a processor, it implements the image acquisition method described above.

[0103] Computer programs can use any combination of one or more storage media. The storage media can be a readable signal medium or a readable storage medium.

[0104] Readable storage media may include, for example, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media may include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0105] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, such as electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any storage medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0106] The computer program contained on the storage medium can be transmitted using any suitable medium, such as wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0107] Computer programs for performing the operations of this invention can be written in any combination of one or more programming languages. Programming languages ​​may include object-oriented programming languages—such as Java, C++, etc.—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The computer program may execute entirely on the user's computing device, partially on the user's device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device may be connected to the user's computing device via any type of network (e.g., including a local area network or a wide area network), or it may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0108] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application. When the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0109] It should be noted that the terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate.

[0110] It should be understood that the exemplary embodiments described herein can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. These embodiments are provided so that the disclosure of this application is thorough and complete, and that the concept of these exemplary embodiments is fully conveyed to those skilled in the art, and should not be construed as limiting the invention.

[0111] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An image acquisition system, characterized in that, The image acquisition system, used in 3D displays, includes: The first image acquisition device is located at the top center of the screen and is used to acquire at least two depth images of the target object in front of the screen. Multiple second image acquisition devices are positioned at the four corners of the screen to acquire images of the target object in front of the screen from multiple different angles; The controller is configured to determine the displacement of the target object based on the at least two frames of depth images, and to control the plurality of second image acquisition devices to deflect accordingly to follow the target object, so that the target object is at least within the common field of view of the plurality of second image acquisition devices, and to control the plurality of second image acquisition devices to acquire images of the target object from multiple different angles; each second image acquisition device can deflect accordingly to follow the target object based on the movable range of the optical axis of the first image acquisition device and the movable range of the optical axes of the plurality of second image acquisition devices. The controller is further configured to: for each frame of depth image, acquire the pixel coordinates and depth value of the target object, and determine the three-dimensional coordinates of the target object in the three-dimensional coordinate system of the first image acquisition device based on the pixel coordinates and depth value of the target object, including: for a point A on the depth image, its pixel coordinates are (ua, va), and its depth value is |Z a First, calculate the projection length of point A on the z-axis of the three-dimensional coordinate system Oc of the first image acquisition device. This projection is equal to the distance from the origin of the three-dimensional coordinate system Oc to the image center Ouv(u0, v0). c_a When the plane containing the depth image is perpendicular to the z-axis of the three-dimensional coordinate system of the first image acquisition device, according to the Pythagorean theorem, the projection length of point A on the z-axis of the three-dimensional coordinate system of the first image acquisition device is... ,in, D represents the distance from point A to the image center Ouv; when the plane containing the depth image is not perpendicular to the z-axis of the three-dimensional coordinate system of the first image acquisition device, the angle between the plane containing the depth image and the xy-plane of the three-dimensional coordinate system of the first image acquisition device is included in the calculation to obtain the projection of point A on the z-axis of the three-dimensional coordinate system; the three-dimensional coordinates P of point A in the three-dimensional coordinate system of the first image acquisition device are determined by the following formula. A (X c_a Y c_a Z c_a ): , where K 1 Z represents the inverse matrix of the intrinsic parameters of the first image acquisition device. c_a Represents a scaling factor; The displacement of the target object is determined based on the three-dimensional coordinates of the target object corresponding to each of the at least two depth images: based on the two three-dimensional coordinates P of point A in the three-dimensional coordinate system of the first image acquisition device corresponding to the two depth images. A new and P A old, obtain the displacement M of point A = (PA). new) (PA old); The controller is further configured to: determine the deflection angle of each of the plurality of second image acquisition devices according to the displacement; control the plurality of second image acquisition devices to deflect according to their respective deflection angles; for each second image acquisition device, determine the displacement projection in the three-dimensional coordinate system of the second image acquisition device based on the pre-calibrated extrinsic parameters of the second image acquisition device relative to the three-dimensional coordinate system of the first image acquisition device; and determine the deflection angle of the second image acquisition device according to the normal vector of the three-dimensional coordinate system of the second image acquisition device and the displacement projection.

2. The image acquisition system according to claim 1, characterized in that, The horizontal field of view of the first image acquisition device is not less than 120°, and / or its vertical field of view is not less than 90°.

3. The image acquisition system according to claim 1, characterized in that, Also includes: A first gimbal, which is mechanically connected to the first image acquisition device, to drive the first image acquisition device to deflect. The controller is further configured to control the movement of the first gimbal based on the displacement, so that the first gimbal drives the first image acquisition device to follow the target object and deflect.

4. The image acquisition system according to claim 1, characterized in that, Also includes: Multiple second gimbals are mechanically connected to multiple second image acquisition devices in a one-to-one correspondence, so as to drive the multiple second image acquisition devices to deflect respectively; The controller is also configured to control the movement of the plurality of second gimbals according to the displacement of the target object, so that the plurality of second gimbals drive the plurality of second image acquisition devices to deflect respectively following the target object.

5. The image acquisition system according to claim 4, characterized in that, Each second gimbal includes at least three rotational degrees of freedom.

6. The image acquisition system according to claim 1, characterized in that, The target object includes the target human body; The controller is further configured to sequentially perform grayscale and binarization processing on each frame of depth image to obtain at least two binarized images, and extract the contour images of the target human body in the at least two binarized images respectively, select a reference point in the contour image, and use the position change of the reference point in the at least two frames of depth images as the displacement of the target object.

7. The image acquisition system according to claim 6, characterized in that, The reference point includes the center point of the neck of the target human body.

8. The image acquisition system according to claim 1, characterized in that, The controller is also used for, The deflection angle of the second image acquisition device is determined based on the fact that the displacement of the normal vector of the three-dimensional coordinate system along the displacement projection is equivalent to the rotation of the normal vector around each coordinate axis.

9. The image acquisition system according to claim 1, characterized in that, The controller is also used for, The extrinsic parameters of the plurality of second image acquisition devices relative to the first image acquisition device are updated based on the deflection angle of each of the plurality of second image acquisition devices.

10. The image acquisition system according to claim 1, characterized in that, The controller is also used for, The displacement distance of the target object is compared with a preset distance threshold. If the displacement distance of the target object exceeds the preset distance threshold, the multiple second image acquisition devices are controlled to follow the target object and deflect accordingly.

11. The image acquisition system according to claim 1, characterized in that, The controller is also used for, A three-dimensional image of the target object is generated using images of the target object from multiple different angles.

12. An image acquisition method, characterized in that, The image acquisition system based on any one of claims 1 to 11 includes: Acquire at least two depth images of the target object; The displacement of the target object is determined based on the at least two depth images, and the plurality of second image acquisition devices are controlled to follow the target object and deflect accordingly, so that the target object is at least within the common field of view of the plurality of second image acquisition devices; The multiple second image acquisition devices are controlled to acquire images of the target object from multiple different angles.

13. A computing device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the image acquisition method as described in claim 12.

14. A storage medium storing a computer program that, when executed by a processor, implements the image acquisition method as described in claim 12.

Images