Multi-camera layout scheme generation method, electronic equipment, storage medium and program product

Abstract

The invention relates to a multi-camera layout scheme generation method, electronic equipment, a storage medium and a program product, and the method comprises the steps: obtaining the size information of a target site of a to-be-arranged camera, and generating a three-dimensional point cloud of the target site based on the size information; obtaining the number of cameras to be arranged and camera internal parameters, and generating a plurality of virtual cameras at preset positions of the three-dimensional point cloud based on the number of cameras and the camera internal parameters; iteratively optimizing the orientation of each virtual camera in the plurality of virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the plurality of virtual cameras; and determining a multi-camera layout scheme of the target site based on the target orientation of each virtual camera in the plurality of virtual cameras. Therefore, the multi-camera layout scheme which has no camera blind area and is high in camera utilization rate of the target site can be automatically and efficiently generated.

Description

Technical Field

[0001] The present disclosure relates to the field of virtual shooting technology, and in particular to a multi-camera layout solution generation method, electronic equipment, storage medium and program product. Background Art

[0002] Determining the position and posture of objects in the field is a common need in many application scenarios. For example, in virtual shooting, aligning the spatial positions of the virtual camera and the real camera is an important prerequisite for realizing the fusion of virtual scenes and real scenes. Therefore, in actual shooting, it is necessary to obtain the position and posture of the real camera in real time and use it to drive the virtual camera. Usually, the cameras used in film and television shooting do not have the function of self-positioning, so it is necessary to use an external optical positioning system to realize the positioning of the camera, that is, it is necessary to use the positioning system to obtain the spatial position and posture of the camera in real time, and multi-camera positioning technology is a more general and reliable solution to solve the optical positioning problem. Multi-camera positioning technology is a method of determining the position and posture of a certain point or multiple points in an object or environment through the collaborative work of multiple cameras. This technology is often used in computer vision, robotics, augmented reality and unmanned driving. The core idea of ​​multi-camera positioning is to use the information of different perspectives captured by multiple cameras, and fuse them through geometry and image processing technology to achieve relatively more accurate and robust position and posture estimation.

[0003] Therefore, this type of multi-camera positioning technology requires the installation of multiple cameras at different locations in the shooting site. These cameras can identify specific luminous points (positioning points) in the site. The recognition results of multiple cameras are fused to calculate the spatial positions of these luminous points to achieve the purpose of tracking and positioning. Among them, in the multi-camera positioning system, the positioning point (or positioning mark, feature point, positioning tag, etc.) refers to a specific point in space used to determine the position and posture of an object or scene. These positioning points can be features that naturally exist in the environment (such as corners, textures, edges, etc.), or they can be artificially added marks (such as markers, targets, reflective balls, etc.). By capturing the positions of these positioning points from different camera perspectives, computer vision and geometric algorithms can be used to accurately locate and estimate the posture of objects.

[0004] Since the venue of some scenes (such as virtual shooting) is very large (20-50 meters), where to install the camera and which area each camera faces directly affects the accuracy of the final positioning. In the past, these decisions were made based on manual experience to determine how many cameras are needed for each shooting site, where each camera should be installed and which direction it should face. However, compared with general positioning scenes, the venues of scenes such as virtual shooting are larger and more complex. Due to the lack of intuitive reference, the venues designed and arranged manually often have the problem of camera blind spots. Blind spots refer to areas that cannot be covered or monitored by cameras. This is due to reasons such as viewing angle, installation position or object occlusion. Some areas may not be captured by one or more cameras, which forms a visual blind spot. In these blind spots, although the target objects may exist, the positioning system cannot accurately identify or locate these objects due to lack of visual information. Therefore, in order to reduce the impact of blind spots, it is usually necessary to carefully plan the position and angle (i.e., direction) of the camera, or use more cameras to ensure more comprehensive coverage, which requires repeated manual adjustments and wastes a lot of time; or although more cameras are added, the camera utilization rate is low, because some cameras are installed but do not play a role in improving tracking quality, wasting camera resources. Summary of the invention

[0005] In view of this, the present disclosure proposes a multi-camera layout solution generation method, electronic device, storage medium and program product, which can automatically and efficiently generate a multi-camera layout solution with no camera blind spots and high camera utilization for a target site.

[0006] According to one aspect of the present disclosure, a method for generating a multi-camera layout solution is provided, comprising: obtaining size information of a target site of cameras to be arranged, and generating a three-dimensional point cloud of the target site based on the size information; obtaining the number of cameras and camera intrinsic parameters of the cameras to be arranged, and generating a plurality of virtual cameras at preset positions of the three-dimensional point cloud based on the number of cameras and the camera intrinsic parameters; iteratively optimizing the orientation of each virtual camera in the plurality of virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the plurality of virtual cameras; wherein the target loss function represents a spatial point loss and a camera diversity loss, the spatial point loss represents the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the plurality of virtual cameras and the orientation of the remaining virtual cameras; and determining a multi-camera layout solution for the target site based on the target orientation of each virtual camera in the plurality of virtual cameras.

[0007] In a possible implementation, the iterative optimization of the orientation of each virtual camera in the multiple virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the multiple virtual cameras includes: iteratively performing the following optimization process based on the target loss function: in the i-th optimization process, according to the i-1th generation orientation of each virtual camera obtained by the i-1th optimization, determining the pixel position of the projection point in the imaging picture of each virtual camera projected from each spatial point in the three-dimensional point cloud, and determining the similarity between the i-1th generation orientation of each virtual camera in the multiple virtual cameras and the i-1th generation orientation of the remaining virtual cameras; wherein, i≥1, the 0th generation orientation is a preset initial orientation; according to the projection of each spatial point in the three-dimensional point cloud to the imaging picture of each virtual camera, The i-th generation spatial point loss is determined by summing the distances between the pixel positions of the projection points in the plane and the pixel positions of the centers of the imaging pictures of the respective virtual cameras; the i-th generation camera diversity loss is determined according to the normalized mean of the cosine similarities between the i-1th generation orientations of each virtual camera in the plurality of virtual cameras and the i-1th generation orientations of the remaining virtual cameras; when the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, the i-1th generation orientations of the respective virtual cameras are optimized based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientations of the respective virtual cameras; when the i-th generation spatial point loss and the i-th generation camera diversity loss have both converged, the i-th generation orientations of the respective virtual cameras are determined as the target orientations of the respective virtual cameras.

[0008] In a possible implementation, the multi-camera layout scheme of the target site includes the orientation and position of each real camera corresponding to each virtual camera; wherein the orientation of each real camera is the target orientation of the corresponding virtual camera, and the position of each real camera corresponds to the preset position.

[0009] In a possible implementation manner, the preset positions include uniformly distributed positions on a specified outer contour of the three-dimensional point cloud.

[0010] In a possible implementation manner, the designated outer contour includes an outer contour of the highest height of the three-dimensional point cloud.

[0011] According to another aspect of the present disclosure, a method for generating a multi-camera layout solution is provided, including: obtaining size information of a target site of cameras to be arranged, and generating a three-dimensional point cloud of the target site based on the size information; obtaining a maximum number of cameras M and camera intrinsic parameters of the cameras to be arranged, and iteratively performing the following generation process in a manner of decreasing one by one from the maximum number of cameras M: in the jth generation process, based on the number of cameras N and the corresponding camera intrinsic parameters, generating N virtual cameras at preset positions of the three-dimensional point cloud; wherein j≥0, N=Mj; iteratively optimizing the orientation of each virtual camera in the N virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the N virtual cameras; wherein the target loss function table The invention relates to a method for detecting a target site in a three-dimensional point cloud. The method comprises the following steps: first, detecting a target site in a three-dimensional point cloud by detecting a spatial point loss and a camera diversity loss. The spatial point loss represents the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging picture after being projected onto the imaging picture of each virtual camera. The camera diversity loss represents the similarity between the orientation of any virtual camera among the N virtual cameras and the orientation of the remaining virtual cameras. When the loss value suddenly increases when the target loss function converges, the multi-camera layout scheme when the number of cameras is N+1 is determined as the target multi-camera layout scheme of the target site. The multi-camera layout scheme is determined according to the target orientation of each virtual camera among the N+1 virtual cameras. The loss value when the target loss function converges includes the loss value when the target orientation of each virtual camera among the N virtual cameras is determined.

[0012] In a possible implementation, the iterative optimization of the orientation of each virtual camera in the N virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the N virtual cameras includes: iteratively performing the following optimization process based on the target loss function: in the i-th optimization process, according to the i-1th generation orientation of each virtual camera obtained by the i-1th optimization, determining the pixel position of the projection point in the imaging picture of each virtual camera projected from each spatial point in the three-dimensional point cloud, and determining the cosine similarity between the i-1th generation orientation of each virtual camera in the N virtual cameras and the i-1th generation orientation of the remaining virtual cameras; wherein, i≥1, the 0th generation orientation is a preset initial orientation; according to the projection of each spatial point in the three-dimensional point cloud to the imaging picture of each virtual camera The i-th generation spatial point loss is determined by the sum of the distances between the pixel position of the projection point in the picture and the pixel position of the center of the imaging picture of each virtual camera; the i-th generation camera diversity loss is determined according to the normalized mean of the cosine similarities between the i-1th generation orientation of each virtual camera in the N virtual cameras and the i-1th generation orientation of the remaining virtual cameras; when the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, the i-1th generation orientation of each virtual camera is optimized based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera; when the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

[0013] In one possible implementation, the multi-camera layout solution when the number of cameras is N+1 includes the orientation and position of each real camera among the N+1 real cameras corresponding to the N+1 virtual cameras; wherein the orientation of each real camera is the target orientation of the corresponding virtual camera, and the position of each real camera corresponds to the preset position.

[0014] In a possible implementation manner, the preset positions include uniformly distributed positions on a specified outer contour of the three-dimensional point cloud.

[0015] In a possible implementation manner, the designated outer contour includes an outer contour of the highest height of the three-dimensional point cloud.

[0016] According to another aspect of the present disclosure, a multi-camera layout solution generation device is provided, comprising: a point cloud generation module, used to obtain size information of a target site of cameras to be arranged, and generate a three-dimensional point cloud of the target site based on the size information; a virtual camera generation module, used to obtain the number of cameras to be arranged and camera internal parameters, and generate a plurality of virtual cameras at preset positions of the three-dimensional point cloud based on the number of cameras and the camera internal parameters; an orientation optimization module, used to iteratively optimize the orientation of each virtual camera in the plurality of virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the plurality of virtual cameras; wherein the target loss function represents a spatial point loss and a camera diversity loss, the spatial point loss represents the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the plurality of virtual cameras and the orientation of the remaining virtual cameras; a solution generation module, used to determine the multi-camera layout solution of the target site based on the target orientation of each virtual camera in the plurality of virtual cameras.

[0017] According to another aspect of the present disclosure, a multi-camera layout solution generation device is provided, including: a three-dimensional point cloud generation module, used to obtain size information of a target site of cameras to be arranged, and generate a three-dimensional point cloud of the target site based on the size information; an iterative generation module, used to obtain a maximum number of cameras M and camera intrinsic parameters of the cameras to be arranged, and iteratively perform the following generation process in a manner of decreasing one by one from the maximum number of cameras M: in the jth generation process, based on the number of cameras N and the corresponding camera intrinsic parameters, N virtual cameras are generated at preset positions of the three-dimensional point cloud; wherein j≥0, N=Mj; based on a target loss function, the orientation of each virtual camera in the N virtual cameras is iteratively optimized to obtain a target orientation of each virtual camera in the N virtual cameras; wherein the target loss function The loss function represents the spatial point loss and the camera diversity loss, wherein the spatial point loss represents the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging picture after being projected onto the imaging picture of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera among the N virtual cameras and the orientation of the remaining virtual cameras; a layout scheme determination module is used to determine the multi-camera layout scheme when the number of cameras is N+1 as the target multi-camera layout scheme of the target site when a loss value suddenly increases when the target loss function converges. The multi-camera layout scheme is determined according to the target orientation of each virtual camera among the N+1 virtual cameras, and the loss value when the target loss function converges includes the loss value when the target orientation of each virtual camera among the N virtual cameras is determined.

[0018] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0019] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0020] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0021] According to various aspects of the present disclosure, by constructing a three-dimensional point cloud of the target site according to the size information of the target site, and generating multiple virtual cameras with a certain number of cameras at preset positions of the three-dimensional point cloud, and then iteratively optimizing the orientation of the virtual camera based on the spatial point loss and the camera diversity loss, a multi-camera layout scheme with no camera blind spots and high camera utilization rate for the target site can be automatically and efficiently generated, that is, the orientations and positions of multiple real cameras to be arranged in the target site are automatically generated, and after the camera layout is performed using the generated multi-camera layout scheme, multiple real cameras can be made to have no blind spots when performing application scenarios such as spatial positioning or target tracking, and have a high camera utilization rate.

[0022] According to other aspects of the present disclosure, a multi-camera layout scheme with different numbers of cameras can be generated by iteratively decreasing from the maximum number of cameras M, so as to automatically and efficiently determine the minimum number of cameras required for the target site to have no camera blind spots and the orientation and position of each real camera with the minimum number of cameras, and have a high camera utilization rate.

[0023] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0025] Figure 1 A flowchart of a method for generating a multi-camera layout solution according to an embodiment of the present disclosure is shown.

[0026] Figure 2a A schematic diagram of a plane outline of a virtual shooting location according to an embodiment of the present disclosure is shown.

[0027] Figure 2b A schematic diagram of a three-dimensional point cloud according to an embodiment of the present disclosure is shown.

[0028] Figure 3 A schematic diagram showing a multi-camera layout solution according to an embodiment of the present disclosure is shown.

[0029] Figure 4 A flowchart of a method for generating a multi-camera layout solution according to an embodiment of the present disclosure is shown.

[0030] Figure 5 A block diagram of a multi-camera layout solution generating device according to an embodiment of the present disclosure is shown.

[0031] Figure 6 A block diagram of a multi-camera layout solution generating device according to an embodiment of the present disclosure is shown.

[0032] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0034] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0035] The term "and / or" herein is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent any one or more elements selected from the set consisting of A, B, and C. In the description of the present disclosure, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0036] It should be understood that the terms "first", "second", etc. in the claims, specifications and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0037] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0038] As mentioned above, in the layout scheme of the multi-camera positioning system in scenes such as virtual shooting, the number of cameras to be used in the venue, the position and orientation of each camera are determined manually based on experience in the past. Different from the traditional multi-camera positioning scene, the venue of the virtual shooting scene is usually larger and more complex, and the difficulty of positioning system layout design is also higher. Due to the lack of effective reference for manual design, the designed layout scheme may not be able to fully cover the shooting venue, and there are blind spots. It is often necessary to adjust it many times after the layout is completed, which wastes a lot of time; sometimes in order to avoid blind spots, more cameras than actual needs are needed, which makes the utilization rate of some cameras lower, resulting in resource waste. Therefore, this scheme provides an automated multi-camera layout method generation method, which can automatically generate the installation position and orientation of each camera in multiple cameras according to the size information of the target venue (such as the virtual shooting venue), and can also generate the number of cameras required for the target venue, and can ensure that there are no camera blind spots in the venue, so that the efficiency of the venue camera layout is higher and the tracking stability is more guaranteed.

[0039] The multi-camera layout scheme generation method of the embodiment of the present disclosure can be deployed on various terminal devices through software or hardware modification. The terminal device involved in the embodiment of the present disclosure may refer to a device with a wireless connection function and / or a wired connection function. The wireless connection function refers to the ability to connect to other devices through wireless connection methods such as wifi and Bluetooth. The terminal device involved in the embodiment of the present disclosure may also communicate with other devices through a wired connection function. The terminal device involved in the embodiment of the present disclosure may be a touch screen, a non-touch screen, or a screenless terminal device. The touch screen terminal device can be controlled by clicking and sliding on the display screen with a finger or a stylus. The non-touch screen device can be connected to an input device such as a mouse, keyboard, touch panel, etc., and the terminal device can be controlled through the input device. For example, a device without a screen may be a Bluetooth speaker without a screen. For example, the terminal device of the present application may include but is not limited to user equipment (UE), mobile device, user terminal, terminal, handheld device, tablet computer, laptop computer, PDA, computing device, etc.

[0040] The multi-camera layout scheme generation method of the embodiment of the present disclosure can also be deployed on a server. The server can be located in the cloud or locally. It can be a physical device or a virtual device, such as a virtual machine, a container, etc., and has a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. It can refer to a device with a wireless connection function. The wireless connection function refers to the ability to connect to other servers or terminal devices through wireless connection methods such as Wi-Fi and Bluetooth. The server involved in the embodiment of the present disclosure may also have the function of communicating through a wired connection. For example, the server of the embodiment of the present disclosure can be located in the cloud, communicate with the terminal device, receive the size information, number of cameras and camera internal parameters of the target site sent by the terminal device, and use the multi-camera layout scheme generation method deployed on the server to generate a multi-camera layout scheme for the target site based on the size information, number of cameras and camera internal parameters, and return it to the terminal device to display the multi-camera layout scheme of the target site to the user in the terminal device.

[0041] Figure 1 A flowchart of a method for generating a multi-camera layout solution according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method includes: step S11 to step S14.

[0042] In step S11, the size information of the target site where the cameras are to be arranged is obtained, and a three-dimensional point cloud of the target site is generated based on the size information.

[0043] The target site may be any site where multiple cameras are to be deployed, such as a virtual shooting site, which is not limited in the embodiments of the present disclosure. The size information of the target site may include, for example, the length, width, height, radius, etc. of the target site. Figure 2a A schematic diagram of the plane outline of a virtual shooting location is shown. The virtual shooting location is a U-shaped area consisting of a semicircular area (or arc area) surrounded by an LED screen and a rectangular area in front of the screen. Therefore, the size information of the virtual shooting location may include the radius of the LED screen area, the width of the rectangular area, and the height of the screen. That is, these three information can be used to define the outline of the virtual shooting location.

[0044] Then, the target site can be represented in the form of a three-dimensional discrete point cloud according to the size information of the target site. For example, Figure 2a The virtual shooting site shown can be obtained Figure 2b The three-dimensional point cloud of the virtual shooting site is shown, wherein: Figure 2b The green point can be a spatial point (or sample point) in a three-dimensional point cloud. A three-dimensional point cloud is a set of points in a three-dimensional space, each of which contains its coordinates in the space (usually X, Y, and Z coordinates), and the distribution of the spatial points can be set as needed. It should be understood that those skilled in the art can use a three-dimensional point cloud generation method known in the art to generate a three-dimensional point cloud of the target site based on the size information of the target site, and the embodiments of the present disclosure are not limited to this.

[0045] In step S12, the number of cameras to be arranged and the camera internal parameters are obtained, and based on the number of cameras and the camera internal parameters, a plurality of virtual cameras are generated at preset positions of the three-dimensional point cloud.

[0046] Among them, the number of cameras can be understood as the number of real cameras that the user expects to be deployed to the target site, and the camera intrinsic parameters can be understood as the camera intrinsic parameters of the real cameras to be deployed to the target site. The camera intrinsic parameters are also the intrinsic parameter matrix of the real camera, and the intrinsic parameter matrix includes parameters such as focal length and optical center offset. It should be understood that a corresponding virtual camera can be generated based on the camera intrinsic parameters, or in other words, the virtual camera uses the same intrinsic parameter matrix as the real camera. Among them, the camera intrinsic parameters of each real camera to be deployed to the target site can be the same or different, and the embodiments of the present disclosure are not limited to this. In addition, the camera mentioned in the embodiments of the present disclosure can be a device for shooting, such as a video camera and a camera, which depends on the camera device used in the target site in actual conditions. The embodiments of the present disclosure do not limit the type and model of the camera.

[0047] It should be understood that, given the number of cameras and the camera intrinsic parameters, a plurality of virtual cameras corresponding to the number of cameras and having the corresponding camera intrinsic parameters can be generated, and the generated plurality of virtual cameras can be distributed on preset positions of the three-dimensional point cloud, and the preset positions can indicate the installation positions of the plurality of real cameras in the target site. Optionally, the preset positions include positions evenly distributed on a specified outer contour of the three-dimensional point cloud, thereby making it possible to evenly distribute the plurality of virtual cameras on the specified outer contour of the three-dimensional point cloud, wherein the specified outer contour can be the outer contour of the target site where the user expects the real cameras to be arranged, for example, the specified outer contour can be the outer contour of the highest altitude of the three-dimensional point cloud, that is, the outer contour of the highest altitude of the target site. By generating a plurality of evenly distributed virtual cameras on the outer contour of the highest altitude of the three-dimensional point cloud (for example, the outer contour of the highest plane (top surface) of the three-dimensional point cloud), the real cameras evenly arranged in the target site can have a better field of view at the highest altitude.

[0048] For example, if the number of cameras is 29, generating 29 virtual cameras evenly distributed on the outer contour of the highest height of the three-dimensional point cloud can be expressed as follows: Figure 2b The distribution of the blue points, that is, Figure 2b The points in medium blue may be 29 virtual cameras evenly distributed and generated on the outer contour of the highest altitude of the three-dimensional point cloud of the target site.

[0049] Among them, each virtual camera generated evenly distributed on the specified outer contour of the three-dimensional point cloud of the target site can have an initial orientation (which can be understood as an initial posture). For example, the initial orientation of each virtual camera can be set by default to be toward the center of the three-dimensional point cloud, which is equivalent to setting the initial orientation to be toward the center of the target site. The orientation can be expressed as the direction vector of the camera, and the direction vector can be concretely represented as a line from the camera position to the camera target point. The camera target point can be understood as the spatial point corresponding to the center of the camera's imaging screen.

[0050] It should be understood that based on multiple virtual cameras evenly distributed on the specified outer contour of the three-dimensional point cloud, the positions of each real camera to be deployed in the target site can be obtained, that is, the positions of each real camera to be deployed in the target site include the positions of each virtual camera evenly distributed on the specified outer contour of the three-dimensional point cloud, which is equivalent to obtaining the actual spatial position of each real camera to be deployed in the target site. Furthermore, in order to ensure that each real camera has no blind spot in the target site, it is also necessary to determine the target orientation (which can be understood as posture) of each real camera through step S13 to obtain a multi-camera layout solution without tracking blind spots in the target site.

[0051] In step S13, the orientation of each virtual camera in the multiple virtual cameras is iteratively optimized based on the target loss function to obtain the target orientation of each virtual camera in the multiple virtual cameras.

[0052] Among them, the target loss function represents the coverage of the target site by the virtual camera and the utilization rate of the virtual camera. The optimization goal is to hope that the coverage rate of the target site by the virtual camera is as high as possible and the utilization rate of the virtual camera is as high as possible. The target loss function can include spatial point loss and camera diversity loss; spatial point loss represents the sum of the degree of deviation from the center of the imaging screen after each spatial point in the three-dimensional point cloud is projected onto the imaging screen of each virtual camera. Among them, the smaller the spatial point loss, the more cameras can cover (i.e., photograph) the spatial point. In order to ensure that there is no tracking blind spot in the target site, it is expected that each spatial point in the three-dimensional point cloud can be covered by more virtual cameras, which is equivalent to expecting that the positioning points in the target site can be covered by more real cameras. Therefore, using spatial point loss to optimize the orientation of the virtual camera can make the optimized target orientation produce as few blind spots as possible.

[0053] Among them, the camera diversity loss represents the similarity between the orientation of any virtual camera among multiple virtual cameras and the orientation of the other virtual cameras. The smaller the camera diversity loss, the greater the difference in orientation between the cameras and the higher the camera utilization rate. Therefore, by using the camera diversity loss to optimize the orientation of the virtual camera, it is possible to avoid that two or more cameras have the same or similar effects in tracking and positioning, causing some cameras to do useless work and waste resources, which is conducive to improving camera utilization.

[0054] In practical applications, optimization algorithms known in the art, such as gradient descent algorithms, optimization algorithms, etc., can be used to iteratively optimize the orientation of each virtual camera in multiple virtual cameras based on the target loss function. For example, the orientation of the camera can be used as an optimization parameter, and the above-mentioned target loss function (i.e., spatial point loss and camera diversity loss) can be used as the optimization target for gradient descent optimization. It is iterated until the target loss function converges (i.e., iterate until the spatial point loss and camera diversity loss no longer change), and the loss value of the target loss function at this time (i.e., the loss value of the spatial point loss and the camera diversity loss) is recorded, and the orientation of each virtual camera at this time is used as the target orientation of the virtual camera.

[0055] Specifically, the above step S13, iteratively optimizing the orientation of each virtual camera in the multiple virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the multiple virtual cameras, includes:

[0056] The following optimization process is iteratively performed based on the target loss function:

[0057] In the i-th optimization process, according to the i-1th generation orientation of each virtual camera obtained by the i-1th optimization, the pixel position of the projection point of each spatial point in the three-dimensional point cloud onto the imaging screen of each virtual camera is determined, and the similarity between the i-1th generation orientation of each virtual camera in the multiple virtual cameras and the i-1th generation orientation of the remaining virtual cameras is determined; wherein, i≥1, the 0th generation orientation is the preset initial orientation;

[0058] Determine the i-th generation spatial point loss according to the sum of the distances between the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging screen of each virtual camera and the pixel position of the center of the imaging screen of each virtual camera;

[0059] Determine the i-th generation camera diversity loss according to a normalized mean of similarities between an i-1-th generation orientation of each virtual camera in the plurality of virtual cameras and an i-1-th generation orientation of the remaining virtual cameras;

[0060] When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, based on the i-th generation spatial point loss and the i-th generation camera diversity loss, the i-1-th generation orientation of each virtual camera is optimized to obtain the i-th generation orientation of each virtual camera;

[0061] When the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

[0062] It should be understood that knowing the orientation (i.e., posture) and position of any virtual camera is equivalent to knowing the camera extrinsic parameters of the virtual camera (i.e., the camera extrinsic matrix). Then, based on the camera intrinsic parameters (intrinsic matrix) and the camera extrinsic parameters (extrinsic matrix), and the imaging principle of the reference camera (i.e., the projection principle of three-dimensional points onto a two-dimensional plane), each spatial point in the three-dimensional point cloud can be projected onto the imaging screen of each virtual camera to obtain the projection point of each spatial point onto the imaging screen and the pixel position of the projection point in the imaging screen (i.e., pixel coordinates). Then, the sum of the distances between the pixel position of the projection point in the imaging screen of each virtual camera and the pixel position of the center of the imaging screen of each virtual camera can be calculated to obtain the spatial point loss. The spatial point loss L can be expressed, for example, as formula (1):

[0063]

[0064] Among them, X t,k represents the pixel position of the projection point in the imaging picture of the kth virtual camera projected from the tth spatial point, X O,k represents the pixel position of the center of the imaging screen of the kth virtual camera, d(X t,k ,XO,k ) represents X t,k With X o,k , T represents the total number of spatial points in the 3D point cloud, and K represents the total number of generated virtual cameras (that is, the number of cameras).

[0065] Among them, those skilled in the art can use distance calculation methods known in the art, such as Euclidean distance, to calculate the distance between two pixel positions (that is, the distance between the projection point and the center of the imaging picture), and the embodiments of the present disclosure are not limited to this.

[0066] As described above, the orientation of the virtual camera can be represented as the direction vector of the virtual camera. Thus, in each round of optimization, the similarity between the direction vector of each virtual camera and the direction vectors of the remaining virtual cameras can be calculated first, that is, the similarity between the orientations of the virtual cameras is calculated two by two, and then the similarity is normalized, and then the mean of the normalized similarity (that is, the normalized mean of the similarity) is calculated to obtain the camera diversity loss. Among them, the similarity between two direction vectors can be, for example, cosine similarity, and of course other similarity algorithms can also be used, which is not limited to the embodiments of the present disclosure. For example, assuming that there are 10 virtual cameras, the cosine similarity between the orientations (direction vectors) of the 10 virtual cameras is calculated two by two, and 45 cosine similarities can be obtained. Then, the 45 cosine similarities can be normalized, and then the mean of the 45 normalized cosine similarities is calculated to obtain the camera diversity loss. Among them, those skilled in the art may adopt any normalization method known in the art, for example, dividing the 45 cosine similarities by the maximum cosine similarity among the 45 cosine similarities or by the average of the 45 cosine similarities to obtain 45 normalized cosine similarities, which is not limited in the embodiments of the present disclosure.

[0067] Among them, those skilled in the art can adopt any optimization algorithm known in the art, such as gradient descent algorithm, optimization algorithm, etc., to optimize the i-1th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss, and obtain the i-th generation orientation of each virtual camera, which is not limited to the embodiments of the present disclosure.

[0068] It should be understood that if the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged (including any one of the i-th generation spatial point loss and the i-th generation camera diversity loss has not converged), then the above-mentioned i-th optimization process can be referred to, and the i+1-th optimization process can be continued based on the i-th generation orientation of each virtual camera until the spatial point loss and the camera diversity loss have converged, and the orientation of each virtual camera optimized when the spatial point loss and the camera diversity loss have converged is determined as the target orientation of each virtual camera. The convergence condition can be selected as needed, for example, when the difference between the values ​​of two or more adjacent spatial point losses is less than a threshold, the spatial point loss is considered to have converged, and when the difference between the values ​​of two or more adjacent camera diversity losses is less than a threshold, the camera diversity loss is considered to have converged.

[0069] In step S14, a multi-camera layout solution of the target site is determined based on the target orientation of each virtual camera in the multiple virtual cameras.

[0070] The multi-camera layout scheme of the target site may include the orientation and position of each real camera corresponding to each virtual camera; wherein the orientation of each real camera is the target orientation of each corresponding virtual camera, and the position of each real camera includes the position of each virtual camera evenly distributed on the specified outer contour of the three-dimensional point cloud. For example, Figure 3 A schematic diagram of a multi-camera layout solution is shown, such as Figure 3 As shown in the figure, the black rectangle represents the real camera (that is, the position of the real camera in the target scene), and the dotted line starting from the black rectangle represents the direction of the camera, that is, the direction of the camera can be visualized as the direction from the camera position to the camera target point (such as Figure 3 The black dots in the figure are connected by a line.

[0071] According to the multi-camera layout scheme generating method of the embodiment of the present disclosure, by constructing a three-dimensional point cloud of the target site according to the size information of the target site, and generating multiple virtual cameras with a certain number of cameras at preset positions of the three-dimensional point cloud, and then iteratively optimizing the orientation of the virtual camera based on the spatial point loss and the camera diversity loss, a multi-camera layout scheme with no camera blind spots and high camera utilization rate for the target site can be automatically and efficiently generated, that is, the orientations and positions of multiple real cameras to be arranged in the target site are automatically generated, and after the generated multi-camera layout scheme is used to perform camera layout, multiple real cameras can be made to have no blind spots when performing application scenarios such as spatial positioning or target tracking, and have a high camera utilization rate.

[0072] Considering that the above Figure 1The multi-camera layout solution generation method shown is a solution generated under a certain number of cameras, and the number of cameras set by the user may not be the optimal number. It is possible to achieve the goal of no blind spots with a smaller number of cameras. Therefore, the embodiment of the present disclosure also provides Figure 4 A flowchart of a method for generating a multi-camera layout solution is shown in FIG. Figure 4 As shown, the method includes: step S41 to step S43.

[0073] In step S41, the size information of the target site where the cameras are to be arranged is obtained, and a three-dimensional point cloud of the target site is generated based on the size information; the implementation of step S41 may refer to the above step S11, which will not be described in detail here.

[0074] In step S42, the maximum number of cameras M and the camera internal parameters of the cameras to be laid out are obtained, wherein the maximum number of cameras can be customized according to actual conditions, which is not limited in the embodiment of the present disclosure; and the following steps S421 to S422 are iteratively executed in a manner of decreasing from the maximum number of cameras M one by one (that is, decreasing from M by one at a time):

[0075] Step S421, in the jth generation process, based on the number of cameras N and the corresponding camera internal parameters, N virtual cameras are generated at the preset positions of the three-dimensional point cloud; wherein j≥0, N=Mj; for example, assuming that the maximum number of cameras M is 29, the 0th generation process generates 29 virtual cameras, the 1st generation process generates 28 virtual cameras, the 2nd generation process generates 27 virtual cameras, and so on;

[0076] Step S422, iteratively optimize the orientation of each virtual camera in the N virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the N virtual cameras; wherein the target loss function represents the spatial point loss and the camera diversity loss, the spatial point loss represents the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the N virtual cameras and the orientation of the remaining virtual cameras.

[0077] The above steps S421 to S422 can be specifically implemented by referring to the above steps S12 to S13, which will not be described in detail. In step S422, the orientation of each virtual camera in the N virtual cameras is iteratively optimized based on the target loss function to obtain the target orientation of each virtual camera in the N virtual cameras, including:

[0078] The following optimization process is iteratively performed based on the target loss function:

[0079] In the i-th optimization process, according to the i-1th generation orientation of each virtual camera obtained by the i-1th optimization, the pixel position of the projection point of each spatial point in the three-dimensional point cloud onto the imaging picture of each virtual camera is determined, and the similarity between the i-1th generation orientation of each virtual camera in the N virtual cameras and the i-1th generation orientation of the remaining virtual cameras is determined; wherein, i≥1, the 0th generation orientation is a preset initial orientation;

[0080] Determine the i-th generation spatial point loss according to the sum of the distances between the pixel position of the projection point in the imaging picture of each virtual camera projected from each spatial point in the three-dimensional point cloud and the pixel position of the center of the imaging picture of each virtual camera;

[0081] Determine the i-th generation camera diversity loss according to the normalized mean of the similarities between the i-1th generation orientation of each virtual camera in the N virtual cameras and the i-1th generation orientation of the remaining virtual cameras;

[0082] When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, optimizing the i-1-th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera;

[0083] When the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

[0084] In step S43, when the loss value suddenly increases when the target loss function converges, the multi-camera layout scheme when the number of cameras is N+1 is determined as the target multi-camera layout scheme for the target site, and the multi-camera layout scheme is determined according to the target orientation of each virtual camera in the N+1 virtual cameras. The loss value when the target loss function converges includes the loss value when the target orientation of each virtual camera in the N virtual cameras is determined.

[0085] Among them, the loss value when the target loss function converges can be understood as the loss value of the space point loss and the camera diversity loss calculated when the target orientation of each virtual camera with N cameras is obtained, that is, the loss value when the space point loss and the camera diversity loss converge; if the loss value when the target loss function corresponding to the number of cameras is N (that is, the space point loss and the camera diversity loss) increases suddenly, it can be considered that the loss value when the target loss function corresponding to the number of cameras is N increases significantly, which means that the target site cannot achieve the goal of no blind spots by using N real cameras, which means that the target site needs to be arranged with at least N+1 real cameras, and N+1 real cameras are the minimum number of cameras that need to be arranged at the target site. Therefore, the multi-camera layout scheme when the number of cameras is N+1 can be determined based on the target orientation of each virtual camera in the N+1 virtual cameras, and the multi-camera layout scheme when the number of cameras is N+1 is determined as the target multi-camera layout scheme for the target site.

[0086] Among them, it is judged whether the loss value suddenly increases when the target loss function converges. For example, it can be judged whether the loss value when the target loss function converges is greater than a preset threshold, and when the loss value when the target loss function converges is greater than the preset threshold, it is considered that the loss value suddenly increases when the target loss function converges. Alternatively, it can also be judged whether the difference between the loss value when the target loss function converges when the number of cameras is N and the loss value when the target loss function converges when the number of cameras is N+1 is greater than the preset threshold, and when the difference is greater than the preset threshold, it is considered that the loss value suddenly increases when the target loss function converges. The embodiments of the present disclosure are not limited to this. It should be understood that the preset threshold can be customized. For example, preset thresholds can be set for the spatial point loss and the camera diversity loss respectively. When any one of the losses in the converged spatial point loss and the camera diversity loss is greater than the corresponding preset threshold, or when the difference between the spatial point loss and the camera diversity loss that converged when the number of cameras is N and the spatial point loss and the camera diversity loss that converged when the number of cameras is N+1 is greater than the preset threshold, it can be considered that the loss value of the target loss function when it converges is greater than the preset threshold; or, a preset threshold can be set for the sum of the spatial point loss and the camera diversity loss. When the sum of the spatial point loss and the camera diversity loss exceeds the preset threshold, or when the difference between the sum of the spatial point loss and the camera diversity loss that converged when the number of cameras is N and the sum of the spatial point loss and the camera diversity loss that converged when the number of cameras is N+1 is greater than the preset threshold, it can be considered that the loss value of the target loss function when it converges is greater than the preset threshold. The embodiments of the present disclosure are not limited to this.

[0087] Among them, the multi-camera layout solution when the number of cameras is N+1 includes the orientation and position of each real camera among the N+1 real cameras corresponding to the N+1 virtual cameras; wherein the orientation of each real camera is the target orientation of the corresponding virtual camera, and the position of each real camera corresponds to the preset position; optionally, the preset position includes the uniformly distributed position on the specified outer contour of the three-dimensional point cloud, that is, the position of each real camera includes the uniformly distributed position of each virtual camera on the specified outer contour of the three-dimensional point cloud, and the specified outer contour includes the outer contour of the highest height of the three-dimensional point cloud.

[0088] The multi-camera layout scheme generation method implemented by the above steps S41 to S43 can be understood as iteratively repeating the above steps S421 to S422 based on the method of reducing the number of cameras by one in each iteration, and the iteration termination condition is that the loss value corresponding to a certain generation of multi-camera layout scheme when the target loss function converges increases significantly, and the number of cameras at this time is recorded as N, and the multi-camera layout scheme when the number of cameras is N+1 is used as the target multi-camera layout scheme used for the target site. For example, if the maximum number of cameras is 29, and the loss value when the target loss function converges when the number of cameras corresponding to the third generation process is 26 is greater than the preset threshold, then the multi-camera layout scheme generated by the second generation process when the number of cameras is 27 can be determined as the target multi-camera layout scheme for the target site. It should be understood that if the loss value when the target loss function converges when the number of cameras corresponding to the third generation process is 26 is still less than or equal to the preset threshold, the generation process of the above steps S421 to S422 can be continued based on the number of cameras being 25 until the loss value when the target loss function converges increases suddenly, so as to obtain the target multi-camera layout plan for the target site.

[0089] According to the multi-camera layout scheme generating method of the embodiment of the present disclosure, by generating multi-camera layout schemes under different numbers of cameras in an iterative manner that decreases one by one from the maximum number of cameras M, it is possible to automatically and efficiently determine the minimum number of cameras required for the target site to have no camera blind spots and the orientation and position of each real camera under the minimum number of cameras, and have a high camera utilization rate.

[0090] According to the multi-camera layout scheme generation method of the embodiment of the present disclosure, it is possible to solve the target multi-camera layout scheme adapted to the current target site in an optimization manner according to the size information of the target site input externally. Compared with manually designing the layout scheme based on experience, the embodiment of the present disclosure directly takes the spatial point loss of the target site and the camera diversity loss as the optimization targets, which can ensure that the final target multi-layout scheme has no tracking blind spots and the camera utilization rate is the highest; the entire process does not require human intervention and can be completed automatically, and the design and generation of the target multi-layout scheme can be completed within seconds, which is more efficient.

[0091] According to the multi-camera layout scheme generating method of the embodiment of the present disclosure, an optimization algorithm is used to solve a target multi-camera layout scheme adapted to the target site. The scheme includes the number of cameras and the pose (i.e., position and orientation) of each camera, and defines and represents the target site in the form of a three-dimensional point cloud. The target loss function is used to quantify the site tracking blind spot and the camera utilization rate, and optimization iterations are performed with this as the goal. This method can better ensure the stability and availability of the multi-camera layout scheme.

[0092] based on Figure 1 The generation method shown is Figure 5 A block diagram of a multi-camera layout solution generating device according to an embodiment of the present disclosure is shown as follows: Figure 5 As shown, the device comprises:

[0093] The point cloud generation module 501 is used to obtain the size information of the target site where the camera is to be arranged, and generate a three-dimensional point cloud of the target site based on the size information;

[0094] A virtual camera generation module 502 is used to obtain the number of cameras to be arranged and camera internal parameters, and generate multiple virtual cameras at preset positions of the three-dimensional point cloud based on the number of cameras and the camera internal parameters;

[0095] The orientation optimization module 503 is used to iteratively optimize the orientation of each virtual camera in the multiple virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the multiple virtual cameras; wherein the target loss function represents the spatial point loss and the camera diversity loss, the spatial point loss represents the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the multiple virtual cameras and the orientation of the remaining virtual cameras;

[0096] The solution generating module 504 is used to determine a multi-camera layout solution for the target site based on the target orientation of each virtual camera in the multiple virtual cameras.

[0097] In a possible implementation, the iteratively optimizing the orientation of each virtual camera in the multiple virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the multiple virtual cameras includes:

[0098] The following optimization process is iteratively performed based on the objective loss function:

[0099] In the i-th optimization process, according to the i-1th generation orientation of each virtual camera obtained by the i-1th optimization, the pixel position of the projection point of each spatial point in the three-dimensional point cloud onto the imaging picture of each virtual camera is determined, and the cosine similarity between the i-1th generation orientation of each virtual camera in the multiple virtual cameras and the i-1th generation orientation of the remaining virtual cameras is determined; wherein, i≥1, the 0th generation orientation is a preset initial orientation;

[0100] Determine the i-th generation spatial point loss according to the sum of the distances between the pixel position of the projection point in the imaging picture of each virtual camera projected from each spatial point in the three-dimensional point cloud and the pixel position of the center of the imaging picture of each virtual camera;

[0101] Determining an i-th generation camera diversity loss according to a normalized mean of cosine similarities between an i-1th generation orientation of each virtual camera in the plurality of virtual cameras and an i-1th generation orientation of the remaining virtual cameras;

[0102] When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, optimizing the i-1th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera;

[0103] When the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

[0104] In one possible implementation, the multi-camera layout scheme of the target site includes the orientation and position of each real camera corresponding to each virtual camera; wherein the orientation of each real camera is the target orientation of the corresponding virtual camera, and the position of each real camera corresponds to the preset position, and the preset position includes a uniformly distributed position on the specified outer contour of the three-dimensional point cloud, and the specified outer contour includes the outer contour of the highest height of the three-dimensional point cloud.

[0105] According to the device of the embodiment of the present disclosure, by constructing a three-dimensional point cloud of the target site according to the size information of the target site, generating multiple virtual cameras with a certain number of cameras at preset positions of the three-dimensional point cloud, and then iteratively optimizing the orientation of the virtual camera based on the spatial point loss and the camera diversity loss, a multi-camera layout scheme for the target site can be automatically and efficiently generated, that is, the orientations and positions of multiple real cameras to be arranged in the target site are automatically generated, and after the camera layout is performed using the generated multi-camera layout scheme, multiple real cameras can be made to have no tracking blind spots when performing application scenarios such as spatial positioning or target tracking, and have a high camera utilization rate.

[0106] based on Figure 4 The generation method shown is Figure 6 A block diagram of a multi-camera layout solution generating device according to an embodiment of the present disclosure is shown as follows: Figure 6 As shown, the device comprises:

[0107] A three-dimensional point cloud generation module 601 is used to obtain size information of a target site where cameras are to be arranged, and to generate a three-dimensional point cloud of the target site based on the size information;

[0108] The iterative generation module 602 is used to obtain the maximum number of cameras M and camera internal parameters to be arranged, and iteratively perform the following generation process in a decreasing manner from the maximum number of cameras M:

[0109] In the jth generation process, based on the number of cameras N and the corresponding camera internal parameters, N virtual cameras are generated at the preset positions of the three-dimensional point cloud; wherein j≥0, N=Mj;

[0110] Iteratively optimizing the orientation of each virtual camera in the N virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the N virtual cameras; wherein the target loss function represents a spatial point loss and a camera diversity loss, the spatial point loss represents the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the N virtual cameras and the orientations of the remaining virtual cameras;

[0111] The layout scheme determination module 603 is used to determine the multi-camera layout scheme when the number of cameras is N+1 as the target multi-camera layout scheme of the target site when the loss value suddenly increases when the target loss function converges. The loss value when the target loss function converges includes the loss value when the target orientation of each virtual camera in the N virtual cameras is determined.

[0112] In a possible implementation, the iterative optimization of the orientation of each virtual camera in the N virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the N virtual cameras includes: iteratively performing the following optimization process based on the target loss function:

[0113] In the i-th optimization process, according to the i-1th generation orientation of each virtual camera obtained by the i-1th optimization, the pixel position of the projection point of each spatial point in the three-dimensional point cloud onto the imaging picture of each virtual camera is determined, and the cosine similarity between the i-1th generation orientation of each virtual camera in the N virtual cameras and the i-1th generation orientation of the remaining virtual cameras is determined; wherein, i≥1, the 0th generation orientation is a preset initial orientation;

[0114] Determine the i-th generation spatial point loss according to the sum of the distances between the pixel position of the projection point in the imaging picture of each virtual camera projected by each spatial point in the three-dimensional point cloud and the pixel position of the center of the imaging picture of each virtual camera;

[0115] Determine the i-th generation camera diversity loss according to the normalized mean of the cosine similarities between the i-1th generation orientation of each virtual camera in the N virtual cameras and the i-1th generation orientation of the remaining virtual cameras;

[0116] When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, optimizing the i-1th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera;

[0117] When the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

[0118] In a possible implementation, the multi-camera layout solution when the number of cameras is N+1 includes the orientation and position of each real camera among the N+1 real cameras corresponding to the N+1 virtual cameras; wherein the orientation of each real camera is the target orientation of the corresponding virtual camera, and the position of each real camera corresponds to the preset position, and the preset position includes uniformly distributed positions on the specified outer contour of the three-dimensional point cloud, and the specified outer contour includes the outer contour of the highest height of the three-dimensional point cloud.

[0119] According to the device of the embodiment of the present disclosure, by generating multi-camera layout solutions under various camera numbers in a manner of decreasing one by one from the maximum camera number M, it is possible to automatically and efficiently determine the minimum number of cameras required for the target site to have no tracking blind spots and the orientation and position of each real camera under the minimum number of cameras, and has a high camera utilization rate.

[0120] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0121] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0122] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0123] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0124] Figure 7 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0125] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0126] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0127] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0128] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0129] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0130] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0131] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0134] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0135] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a multi-camera layout solution, characterized in that: include: Acquire size information of a target site where cameras are to be arranged, and generate a three-dimensional point cloud of the target site based on the size information; Acquire the number of cameras to be laid out and camera internal parameters, and generate multiple virtual cameras at preset positions of the three-dimensional point cloud based on the number of cameras and the camera internal parameters; Iteratively optimizing the orientation of each virtual camera in the multiple virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the multiple virtual cameras; wherein the target loss function represents a spatial point loss and a camera diversity loss, the spatial point loss represents the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the multiple virtual cameras and the orientations of the remaining virtual cameras; A multi-camera layout solution for the target site is determined based on the target orientation of each virtual camera in the multiple virtual cameras.

2. The method according to claim 1, characterized in that The iterative optimization of the orientation of each virtual camera in the multiple virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the multiple virtual cameras includes: The following optimization process is iteratively performed based on the objective loss function: In the i-th optimization process, according to the i-1th generation orientation of each virtual camera obtained by the i-1th optimization, the pixel position of the projection point of each spatial point in the three-dimensional point cloud projected onto the imaging picture of each virtual camera is determined, and the similarity between the i-1th generation orientation of each virtual camera in the multiple virtual cameras and the i-1th generation orientation of the remaining virtual cameras is determined; wherein, i≥1, the 0th generation orientation is a preset initial orientation; Determine the i-th generation spatial point loss according to the sum of the distances between the pixel position of the projection point in the imaging picture of each virtual camera projected by each spatial point in the three-dimensional point cloud and the pixel position of the center of the imaging picture of each virtual camera; Determine the i-th generation camera diversity loss according to a normalized mean of similarities between an i-1-th generation orientation of each virtual camera in the plurality of virtual cameras and an i-1-th generation orientation of the remaining virtual cameras; When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, optimizing the i-1th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera; When the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

3. The method according to claim 1 or 2, characterized in that: The multi-camera layout scheme of the target site includes the orientation and position of each real camera corresponding to each virtual camera; The orientation of each real camera is the target orientation of each corresponding virtual camera, and the position of each real camera corresponds to the preset position.

4. The method according to claim 1 or 2, characterized in that: The preset positions include positions evenly distributed on a specified outer contour of the three-dimensional point cloud.

5. The method according to claim 4, characterized in that The designated outer contour includes an outer contour of the highest height of the three-dimensional point cloud.

6. A method for generating a multi-camera layout solution, characterized in that: include: Acquire size information of a target site where cameras are to be arranged, and generate a three-dimensional point cloud of the target site based on the size information; The maximum number of cameras M and the camera internal parameters to be laid out are obtained, and the following generation process is iteratively performed in a decreasing manner from the maximum number of cameras M: In the jth generation process, based on the number of cameras N and the corresponding camera internal parameters, N virtual cameras are generated at the preset positions of the three-dimensional point cloud; wherein j≥0, N=Mj; Iteratively optimizing the orientation of each virtual camera in the N virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the N virtual cameras; wherein the target loss function represents a spatial point loss and a camera diversity loss, the spatial point loss represents the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the N virtual cameras and the orientations of the remaining virtual cameras; In the case that the loss value suddenly increases when the target loss function converges, the multi-camera layout scheme when the number of cameras is N+1 is determined as the target multi-camera layout scheme for the target site, and the multi-camera layout scheme is determined according to the target orientation of each virtual camera in the N+1 virtual cameras. The loss value when the target loss function converges includes the loss value when the target orientation of each virtual camera in the N virtual cameras is determined.

7. The method according to claim 6, characterized in that The iterative optimization of the orientation of each virtual camera in the N virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the N virtual cameras includes: The following optimization process is iteratively performed based on the objective loss function: In the i-th optimization process, according to the i-1th generation orientation of each virtual camera obtained by the i-1th optimization, the pixel position of the projection point of each spatial point in the three-dimensional point cloud onto the imaging picture of each virtual camera is determined, and the similarity between the i-1th generation orientation of each virtual camera in the N virtual cameras and the i-1th generation orientation of the remaining virtual cameras is determined; wherein, i≥1, the 0th generation orientation is a preset initial orientation; Determine the i-th generation spatial point loss according to the sum of the distances between the pixel position of the projection point in the imaging picture of each virtual camera projected by each spatial point in the three-dimensional point cloud and the pixel position of the center of the imaging picture of each virtual camera; Determine the i-th generation camera diversity loss according to a normalized mean of similarities between the i-1th generation orientation of each virtual camera in the N virtual cameras and the i-1th generation orientations of the remaining virtual cameras; When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, optimizing the i-1th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera; When the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

8. The method according to claim 6 or 7, characterized in that: The multi-camera layout solution when the number of cameras is N+1 includes the orientation and position of each real camera in the N+1 real cameras corresponding to the N+1 virtual cameras; The orientation of each real camera is the target orientation of each corresponding virtual camera, and the position of each real camera corresponds to the preset position.

9. The method according to claim 6 or 7, characterized in that: The preset positions include positions evenly distributed on a specified outer contour of the three-dimensional point cloud.

10. The method according to claim 9, characterized in that The designated outer contour includes an outer contour of the highest height of the three-dimensional point cloud.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 10 when executing the instructions stored in the memory.

12. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer program product, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device executes the method according to any one of claims 1 to 10.

Images