Multi-camera layout scheme generation method, electronic device, storage medium and program product

Abstract

The present disclosure relates to a multi-camera layout scheme generation method, an electronic device, a storage medium and a program product. The method comprises: obtaining size information of a target site where cameras are to be arranged, and generating a three-dimensional point cloud of the target site based on the size information; obtaining a number of cameras to be arranged and camera intrinsic parameters, and generating a plurality of virtual cameras at preset positions of the three-dimensional point cloud based on the number of cameras and the camera intrinsic parameters; iteratively optimizing orientations of each virtual camera in the plurality of virtual cameras based on a target loss function to obtain target orientations of each virtual camera in the plurality of virtual cameras; and determining a multi-camera layout scheme of the target site based on the target orientations of each virtual camera in the plurality of virtual cameras. Thus, a multi-camera layout scheme with no camera blind area and high camera utilization for the target site can be automatically and efficiently generated.

Description

Technical Field

[0001] The present disclosure relates to the field of virtual shooting technology, and in particular to a multi-camera layout solution generation method, electronic equipment, storage medium, and program product. Background Art

[0002] Determining the position and pose of objects in a scene is a common requirement in many application scenarios. For example, in virtual filming, aligning the spatial positions of the virtual camera and the real camera is a crucial prerequisite for achieving a fusion of the virtual and real scenes. Therefore, in real-time filming, it is necessary to obtain the position and pose of the real camera in real time and use this information to drive the virtual camera. Typically, cameras used in film and television production lack self-positioning capabilities, requiring an external optical positioning system to accurately locate the camera. This system, in turn, requires real-time acquisition of the camera's spatial position and pose. Multi-camera positioning technology is a versatile and reliable solution to this optical positioning problem. Multi-camera positioning uses multiple cameras to collaboratively determine the position and pose of one or more points in an object or environment. This technology is commonly used in fields such as computer vision, robotics, augmented reality, and autonomous driving. The core concept of multi-camera positioning is to utilize information captured from different perspectives by multiple cameras and fuse them through geometric and image processing techniques to achieve relatively more accurate and robust position and pose estimation.

[0003] Therefore, this type of multi-camera positioning technology requires the installation of multiple cameras at different locations in the shooting area. These cameras can identify specific light-emitting points (locating points) within the area. The recognition results of multiple cameras are integrated to calculate the spatial position of these light-emitting points, achieving the purpose of tracking and positioning. Among them, in a multi-camera positioning system, the positioning point (also known as positioning mark, feature point, positioning tag, etc.) refers to the specific point in space used to determine the position and posture of an object or scene. These positioning points can be features naturally existing in the environment (such as corners, textures, edges, etc.) or artificially added markers (such as markers, targets, reflective balls, etc.). By capturing the position of these positioning points from different camera perspectives, computer vision and geometric algorithms can be used to accurately locate the object and estimate its posture.

[0004] Because some scenarios (such as virtual filming) involve very large locations (20-50 meters), the location of camera installation and the direction each camera faces directly impact the accuracy of the final positioning. Traditionally, these decisions have relied on manual experience to determine the number of cameras required for each location, as well as the location and orientation of each camera. However, compared to typical positioning scenarios, virtual filming locations are larger and more complex. Manually designed layouts, lacking intuitive references, often result in camera blind spots. A blind spot is an area that cannot be covered or monitored by a camera. Due to factors such as viewing angle, installation position, or obstruction, one or more cameras may not capture this area, creating a visual blind spot. In these blind spots, although the target object may be present, the positioning system cannot accurately identify or locate it due to a lack of visual information. Therefore, to mitigate the impact of blind spots, careful planning of camera positions and angles (i.e., orientations) is often required, or more cameras must be used to ensure more comprehensive coverage. This requires repeated manual adjustments and is time-consuming. Alternatively, adding more cameras may result in low camera utilization, as some cameras may not improve tracking quality, wasting camera resources. Summary of the Invention

[0005] In view of this, the present disclosure proposes a multi-camera layout solution generation method, electronic device, storage medium and program product, which can automatically and efficiently generate a multi-camera layout solution for the target site with no camera blind spots and high camera utilization.

[0006] According to one aspect of the present disclosure, a method for generating a multi-camera layout solution is provided, comprising: obtaining size information of a target site where cameras are to be arranged, and generating a three-dimensional point cloud of the target site based on the size information; obtaining the number of cameras and camera intrinsic parameters of the cameras to be arranged, and generating a plurality of virtual cameras at preset positions of the three-dimensional point cloud based on the number of cameras and the camera intrinsic parameters; iteratively optimizing the orientation of each virtual camera in the plurality of virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the plurality of virtual cameras; wherein the target loss function represents a spatial point loss and a camera diversity loss, the spatial point loss represents the sum of the degrees to which each spatial point in the three-dimensional point cloud deviates from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the plurality of virtual cameras and the orientations of the remaining virtual cameras; and determining a multi-camera layout solution for the target site based on the target orientation of each virtual camera in the plurality of virtual cameras.

[0007] In a possible implementation, the iterative optimization of the orientation of each virtual camera in the multiple virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the multiple virtual cameras includes: iteratively performing the following optimization process based on the target loss function: in the i-th optimization process, according to the i-1th generation orientation of each virtual camera obtained by the i-1th optimization, determining the pixel position of the projection point in the imaging picture of each virtual camera projected on each spatial point in the three-dimensional point cloud, and determining the similarity between the i-1th generation orientation of each virtual camera in the multiple virtual cameras and the i-1th generation orientation of the remaining virtual cameras; wherein, i≥1, the 0th generation orientation is a preset initial orientation; according to the projection of each spatial point in the three-dimensional point cloud onto the imaging picture of each virtual camera, the pixel position of the projection point in the imaging picture of each virtual camera is determined. The i-th generation spatial point loss is determined by summing the distances between the pixel positions of the projection points in the plane and the pixel positions of the centers of the imaging frames of the respective virtual cameras; the i-th generation camera diversity loss is determined according to the normalized mean of the cosine similarities between the i-1th generation orientations of each virtual camera in the plurality of virtual cameras and the i-1th generation orientations of the remaining virtual cameras; if the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, the i-1th generation orientations of the respective virtual cameras are optimized based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientations of the respective virtual cameras; if the i-th generation spatial point loss and the i-th generation camera diversity loss have both converged, the i-th generation orientations of the respective virtual cameras are determined as the target orientations of the respective virtual cameras.

[0008] In one possible implementation, the multi-camera layout scheme of the target site includes the orientation and position of each real camera corresponding to each virtual camera; wherein the orientation of each real camera is the target orientation of the corresponding virtual camera, and the position of each real camera corresponds to the preset position.

[0009] In a possible implementation, the preset positions include evenly distributed positions on a specified outer contour of the three-dimensional point cloud.

[0010] In a possible implementation, the designated outer contour includes an outer contour at a highest height of the three-dimensional point cloud.

[0011] According to another aspect of the present disclosure, a multi-camera layout solution generation method is provided, comprising: obtaining size information of a target site for cameras to be arranged, and generating a three-dimensional point cloud of the target site based on the size information; obtaining a maximum number M of cameras to be arranged and camera intrinsic parameters, and iteratively performing the following generation process in a decreasing manner from the maximum number M of cameras: in the jth generation process, based on the number N of cameras and the corresponding camera intrinsic parameters, generating N virtual cameras at preset positions of the three-dimensional point cloud; wherein j ≥ 0, N = Mj; iteratively optimizing the orientation of each virtual camera in the N virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the N virtual cameras; wherein the target loss function table Characterize spatial point loss and camera diversity loss, the spatial point loss represents the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the N virtual cameras and the orientations of the remaining virtual cameras; in the case where the loss value suddenly increases when the target loss function converges, the multi-camera layout scheme when the number of cameras is N+1 is determined as the target multi-camera layout scheme of the target site, the multi-camera layout scheme is determined according to the target orientation of each virtual camera in the N+1 virtual cameras, and the loss value when the target loss function converges includes the loss value when the target orientation of each virtual camera in the N virtual cameras is determined.

[0012] In a possible implementation, the iterative optimization of the orientation of each virtual camera in the N virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the N virtual cameras includes: iteratively performing the following optimization process based on the target loss function: in the i-th optimization process, according to the i-1th generation orientation of each virtual camera obtained by the i-1th optimization, determining the pixel position of the projection point of each spatial point in the three-dimensional point cloud projected onto the imaging screen of each virtual camera, and determining the cosine similarity between the i-1th generation orientation of each virtual camera in the N virtual cameras and the i-1th generation orientation of the remaining virtual cameras; wherein, i≥1, the 0th generation orientation is a preset initial orientation; according to the projection of each spatial point in the three-dimensional point cloud onto the imaging screen of each virtual camera, the pixel position of the projection point of each virtual camera in the three-dimensional point cloud projected onto the imaging screen of each virtual camera is determined. The i-th generation spatial point loss is determined by the sum of the distances between the pixel position of the projected point in the picture and the pixel position of the center of the imaging picture of each virtual camera; the i-th generation camera diversity loss is determined according to the normalized mean of the cosine similarities between the i-1th generation orientation of each virtual camera in the N virtual cameras and the i-1th generation orientation of the remaining virtual cameras; if the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, the i-1th generation orientation of each virtual camera is optimized based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera; if the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

[0013] In one possible implementation, the multi-camera layout solution when the number of cameras is N+1 includes the orientation and position of each real camera among the N+1 real cameras corresponding to the N+1 virtual cameras; wherein the orientation of each real camera is the target orientation of the corresponding virtual camera, and the position of each real camera corresponds to the preset position.

[0014] In a possible implementation, the preset positions include evenly distributed positions on a specified outer contour of the three-dimensional point cloud.

[0015] In a possible implementation, the designated outer contour includes an outer contour at a highest height of the three-dimensional point cloud.

[0016] According to another aspect of the present disclosure, a multi-camera layout solution generation device is provided, comprising: a point cloud generation module for obtaining dimension information of a target site where cameras are to be arranged, and generating a three-dimensional point cloud of the target site based on the dimension information; a virtual camera generation module for obtaining the number of cameras to be arranged and camera intrinsic parameters, and generating a plurality of virtual cameras at preset positions on the three-dimensional point cloud based on the number of cameras and the camera intrinsic parameters; an orientation optimization module for iteratively optimizing the orientation of each virtual camera in the plurality of virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the plurality of virtual cameras; wherein the target loss function represents a spatial point loss and a camera diversity loss, wherein the spatial point loss represents the sum of the degrees to which each spatial point in the three-dimensional point cloud deviates from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the plurality of virtual cameras and the orientations of the remaining virtual cameras; and a solution generation module for determining a multi-camera layout solution for the target site based on the target orientation of each virtual camera in the plurality of virtual cameras.

[0017] According to another aspect of the present disclosure, a multi-camera layout solution generation device is provided, comprising: a three-dimensional point cloud generation module, for obtaining size information of a target site for cameras to be arranged, and generating a three-dimensional point cloud of the target site based on the size information; an iterative generation module, for obtaining a maximum number M of cameras to be arranged and camera intrinsic parameters, and iteratively performing the following generation process in a manner of decreasing from the maximum number M of cameras one by one: in the j-th generation process, based on the number N of cameras and the corresponding camera intrinsic parameters, generating N virtual cameras at preset positions of the three-dimensional point cloud; wherein j≥0, N=Mj; iteratively optimizing the orientation of each virtual camera in the N virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the N virtual cameras; wherein the target loss function The loss function characterizes the spatial point loss and the camera diversity loss. The spatial point loss characterizes the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging screen after being projected onto the imaging screen of each virtual camera. The camera diversity loss characterizes the similarity between the orientation of any virtual camera among the N virtual cameras and the orientations of the remaining virtual cameras. A layout scheme determination module is used to determine the multi-camera layout scheme when the number of cameras is N+1 as the target multi-camera layout scheme of the target site when a loss value suddenly increases when the target loss function converges. The multi-camera layout scheme is determined according to the target orientation of each virtual camera in the N+1 virtual cameras. The loss value when the target loss function converges includes the loss value when the target orientation of each virtual camera in the N virtual cameras is determined.

[0018] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0019] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0020] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0021] According to various aspects of the present disclosure, by constructing a three-dimensional point cloud of the target site based on the size information of the target site, and generating multiple virtual cameras with a certain number of cameras at preset positions of the three-dimensional point cloud, and then iteratively optimizing the orientation of the virtual cameras based on spatial point loss and camera diversity loss, a multi-camera layout solution with no camera blind spots and high camera utilization can be automatically and efficiently generated for the target site, that is, the orientations and positions of multiple real cameras to be arranged in the target site are automatically generated, and after the cameras are laid out using the generated multi-camera layout solution, the multiple real cameras can be made to have no blind spots when performing application scenarios such as spatial positioning or target tracking, and have high camera utilization.

[0022] According to other aspects of the present disclosure, a multi-camera layout scheme with different numbers of cameras can be generated by iteratively decreasing from the maximum number of cameras M. This can automatically and efficiently determine the minimum number of cameras required to eliminate camera blind spots in the target site and the orientation and position of each real camera with the minimum number of cameras, and has a high camera utilization rate.

[0023] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0025] Figure 1 A flowchart of a method for generating a multi-camera layout solution according to an embodiment of the present disclosure is shown.

[0026] Figure 2a A schematic diagram of a planar outline of a virtual shooting location according to an embodiment of the present disclosure is shown.

[0027] Figure 2b A schematic diagram of a three-dimensional point cloud according to an embodiment of the present disclosure is shown.

[0028] Figure 3 A schematic diagram illustrating a multi-camera layout solution according to an embodiment of the present disclosure is shown.

[0029] Figure 4 A flowchart of a method for generating a multi-camera layout solution according to an embodiment of the present disclosure is shown.

[0030] Figure 5 A block diagram of a multi-camera layout solution generating device according to an embodiment of the present disclosure is shown.

[0031] Figure 6 A block diagram of a multi-camera layout solution generating device according to an embodiment of the present disclosure is shown.

[0032] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0034] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0035] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C. In the description of this disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0036] It should be understood that the terms "first," "second," and the like in the claims, specification, and drawings of the present disclosure are used to distinguish between different objects, rather than to describe a specific order. The terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0037] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0038] As mentioned above, in the past, the layout of multi-camera positioning systems in scenarios such as virtual shooting was manually determined based on experience to determine the number of cameras to be used in the venue, the position and orientation of each camera. Unlike traditional multi-camera positioning scenarios, the venues of virtual shooting scenes are usually larger and more complex, and the difficulty of positioning system layout design is also higher. Due to the lack of effective reference for manual design, the designed layout plan may not fully cover the shooting venue, and there will be blind spots. After the layout is completed, it is often necessary to adjust it multiple times, wasting a lot of time. Sometimes, in order to avoid blind spots, more cameras than actually required are needed, making the utilization rate of some cameras lower and causing resource waste. Therefore, this solution provides an automated multi-camera layout generation method, which can automatically generate the installation position and orientation of each camera in the target venue (such as the virtual shooting venue) according to the size information of the target venue (such as the virtual shooting venue), and can also generate the number of cameras required for the target venue, and can ensure that there are no camera blind spots in the venue, making the venue camera layout more efficient and the tracking stability more guaranteed.

[0039] The multi-camera layout solution generation method of the embodiment of the present disclosure can be deployed on various terminal devices through software or hardware modification. The terminal device involved in the embodiment of the present disclosure may refer to a device with a wireless connection function and / or a wired connection function. The wireless connection function refers to the ability to connect to other devices through wireless connection methods such as wifi and Bluetooth. The terminal device involved in the embodiment of the present disclosure can also communicate with other devices through a wired connection function. The terminal device involved in the embodiment of the present disclosure can be touch-screen, non-touch-screen, or have no screen. The touch-screen terminal device can be controlled by clicking, sliding, etc. on the display screen with a finger or a stylus. The non-touch-screen device can be connected to an input device such as a mouse, keyboard, touch panel, and the terminal device can be controlled through the input device. For example, a device without a screen can be a Bluetooth speaker without a screen. For example, the terminal device of the present application may include but is not limited to user equipment (UE), mobile device, user terminal, terminal, handheld device, tablet computer, laptop computer, PDA, computing device, etc.

[0040] The multi-camera layout solution generation method of the embodiment of the present disclosure can also be deployed on a server. The server can be located in the cloud or locally. It can be a physical device or a virtual device, such as a virtual machine, a container, etc., and has a wireless communication function, wherein the wireless communication function can be set in the chip (system) or other parts or components of the server. It can refer to a device with a wireless connection function. The wireless connection function means that it can be connected to other servers or terminal devices through wireless connection methods such as Wi-Fi and Bluetooth. The server involved in the embodiment of the present disclosure can also have the function of communicating via a wired connection. For example, the server of the embodiment of the present disclosure can be located in the cloud, communicate with the terminal device, receive the size information, number of cameras and camera internal parameters of the target site sent by the terminal device, and use the multi-camera layout solution generation method deployed on the server to generate a multi-camera layout solution for the target site based on the size information, number of cameras and camera internal parameters, and return it to the terminal device to display the multi-camera layout solution of the target site to the user in the terminal device.

[0041] Figure 1 FIG. 1 is a flow chart showing a method for generating a multi-camera layout solution according to an embodiment of the present disclosure. Figure 1 As shown, the method includes: steps S11 to S14.

[0042] In step S11 , the size information of the target site where the cameras are to be arranged is obtained, and a three-dimensional point cloud of the target site is generated based on the size information.

[0043] The target site can be any site where multiple cameras are to be arranged, such as a virtual shooting site, which is not limited in the present embodiment. The size information of the target site can include, for example, the length, width, height, radius and other information of the target site. For example, Figure 2a The figure shows a schematic diagram of the planar outline of a virtual shooting location. The virtual shooting location is a U-shaped area consisting of a semicircular area (or arc-shaped area) surrounded by an LED screen and a rectangular area in front of the screen. Therefore, the size information of the virtual shooting location can include the radius of the LED screen area, the width of the rectangular area, and the height of the screen. In other words, these three pieces of information can be used to define the outline of the virtual shooting location.

[0044] Then, the target site can be represented in the form of a three-dimensional discrete point cloud according to the size information of the target site. For example, Figure 2a The virtual shooting site shown can be obtained Figure 2b The three-dimensional point cloud of the virtual shooting site is shown, wherein Figure 2b The green points can be spatial points (or sample points) in a three-dimensional point cloud. A three-dimensional point cloud is a collection of points in three-dimensional space. Each spatial point contains its coordinates in space (usually X, Y, and Z coordinates). The distribution of spatial points can be set as needed. It should be understood that those skilled in the art can use three-dimensional point cloud generation methods known in the art to generate a three-dimensional point cloud of the target site based on the size information of the target site, and this embodiment of the present disclosure is not limited to this.

[0045] In step S12, the number of cameras to be arranged and the camera intrinsic parameters are obtained, and based on the number of cameras and the camera intrinsic parameters, a plurality of virtual cameras are generated at preset positions of the three-dimensional point cloud.

[0046] Among them, the number of cameras can be understood as the number of real cameras that the user expects to be deployed to the target site, and the camera intrinsic parameters can be understood as the camera intrinsic parameters of the real cameras to be deployed to the target site. The camera intrinsic parameters are also the intrinsic parameter matrix of the real camera, and the intrinsic parameter matrix includes parameters such as focal length and optical center offset. It should be understood that a corresponding virtual camera can be generated based on the camera intrinsic parameters, or in other words, the virtual camera uses the same intrinsic parameter matrix as the real camera. Among them, the camera intrinsic parameters of each real camera to be deployed to the target site can be the same or different, and this embodiment of the present disclosure does not limit this. In addition, the camera mentioned in the embodiment of the present disclosure can be a device for shooting, such as a video camera and a camera, which depends on the camera device used in the target site in actual conditions. The embodiment of the present disclosure does not limit the type and model of the camera.

[0047] It should be understood that, given the number of cameras and the camera intrinsic parameters, a plurality of virtual cameras corresponding to the number of cameras and having the corresponding camera intrinsic parameters can be generated. The generated plurality of virtual cameras can be distributed on preset positions of the three-dimensional point cloud, which can indicate the installation positions of the plurality of real cameras in the target site. Optionally, the preset positions include positions evenly distributed on the specified outer contour of the three-dimensional point cloud, thereby evenly distributing the plurality of virtual cameras to the specified outer contour of the three-dimensional point cloud, wherein the specified outer contour can be the outer contour of the target site where the user expects the real cameras to be arranged. For example, the specified outer contour can be the outer contour of the highest altitude of the three-dimensional point cloud, that is, the outer contour of the highest altitude of the target site. By generating a plurality of evenly distributed virtual cameras on the outer contour of the highest altitude of the three-dimensional point cloud (for example, the outer contour of the highest plane (top surface) of the three-dimensional point cloud), the real cameras evenly distributed in the target site can have a better field of view at the highest altitude.

[0048] For example, if the number of cameras is 29, generating 29 virtual cameras evenly distributed on the outer contour of the highest height of the 3D point cloud can be expressed as follows: Figure 2b The distribution of the blue points, that is, Figure 2b The blue points may be 29 virtual cameras evenly distributed on the outer contour of the highest altitude of the 3D point cloud of the target site.

[0049] Among them, each virtual camera generated evenly distributed on the specified outer contour of the three-dimensional point cloud of the target site can have an initial orientation (which can be understood as an initial posture). For example, the initial orientation of each virtual camera can be set by default to the center of the three-dimensional point cloud, which is equivalent to setting the initial orientation to the center of the target site. The orientation can be expressed as the direction vector of the camera, and the direction vector can be visualized as a line from the camera position to the camera target point. The camera target point can be understood as the spatial point corresponding to the center of the camera's imaging screen.

[0050] It should be understood that based on the multiple virtual cameras evenly distributed on the specified outer contour of the 3D point cloud, the positions of each real camera to be deployed in the target site can be obtained. In other words, the positions of each real camera to be deployed in the target site include the positions of each virtual camera evenly distributed on the specified outer contour of the 3D point cloud, which is equivalent to obtaining the actual spatial position of each real camera to be deployed in the target site. Furthermore, in order to ensure that each real camera has no blind spots in the target site, it is necessary to determine the target orientation (which can be understood as the posture) of each real camera in step S13 to obtain a multi-camera layout solution for the target site without tracking blind spots.

[0051] In step S13 , the orientation of each virtual camera in the multiple virtual cameras is iteratively optimized based on the target loss function to obtain the target orientation of each virtual camera in the multiple virtual cameras.

[0052] The target loss function represents coverage of the target site by the virtual cameras and utilization of the virtual cameras, and the optimization target is to make the coverage of the target site by the virtual cameras as high as possible and the utilization of the virtual cameras as high as possible. The target loss function can include a spatial point loss and a camera diversity loss. The spatial point loss represents a sum of degrees of deviation of each spatial point in the three-dimensional point cloud from a center of an imaging picture after being projected onto imaging pictures of the virtual cameras, where the smaller the spatial point loss is, the more cameras the spatial point can be covered by (i.e., photographed by). In order to ensure that there is no tracking blind area in the target site, it is expected that each spatial point in the three-dimensional point cloud can be covered by more virtual cameras, which is equivalent to expecting that the positioning points in the target site can be covered by more real cameras. Therefore, optimizing the orientations of the virtual cameras by using the spatial point loss can make the target orientations obtained by the optimization have as few blind areas as possible.

[0053] The camera diversity loss represents a similarity between the orientation of any virtual camera in the plurality of virtual cameras and the orientations of the remaining virtual cameras, where the smaller the camera diversity loss is, the greater the difference between the orientations of the cameras is, and the higher the utilization of the cameras is. Therefore, optimizing the orientations of the virtual cameras by using the camera diversity loss can avoid as much as possible the situation that two or more cameras have the same or similar effects in tracking and positioning, so that some cameras do not work and resources are wasted, which is conducive to improving the utilization of the cameras.

[0054] In actual applications, an optimization algorithm known in the art, such as a gradient descent algorithm or an optimization algorithm, can be used to iteratively optimize the orientations of the virtual cameras in the plurality of virtual cameras based on the target loss function. For example, the orientations of the cameras can be used as optimization parameters, and the gradient descent optimization can be performed on the target loss function (i.e., the spatial point loss and the camera diversity loss) to iterate until the target loss function converges (i.e., the spatial point loss and the camera diversity loss no longer change), record the loss value of the target loss function (i.e., the loss values of the spatial point loss and the camera diversity loss) at this time, and use the orientations of the virtual cameras at this time as the target orientations of the virtual cameras.

[0055] Specifically, the step S13 of iteratively optimizing the orientations of the virtual cameras in the plurality of virtual cameras based on the target loss function to obtain the target orientations of the virtual cameras in the plurality of virtual cameras includes:

[0056] The following optimization process is iteratively performed based on the target loss function:

[0057] During the i-th optimization process, determining the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging image of each virtual camera based on the i-1-th generation orientation of each virtual camera obtained during the i-1-th optimization process, and determining the similarity between the i-1-th generation orientation of each virtual camera in the plurality of virtual cameras and the i-1-th generation orientation of the remaining virtual cameras; wherein i ≥ 1, and the generation 0 orientation is a preset initial orientation;

[0058] Determine the i-th generation spatial point loss based on the sum of the distances between the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging screen of each virtual camera and the pixel position of the center of the imaging screen of each virtual camera;

[0059] determining a camera diversity loss of the i-th generation according to a normalized mean of similarities between the i-1-th generation orientation of each virtual camera in the plurality of virtual cameras and the i-1-th generation orientations of the remaining virtual cameras;

[0060] When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, optimize the i-1-th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera;

[0061] When both the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

[0062] It should be understood that knowing the orientation (i.e., posture) and position of any virtual camera is equivalent to knowing the camera extrinsic parameters of the virtual camera (i.e., the camera extrinsic matrix). Then, based on the camera intrinsic parameters (intrinsic matrix) and the camera extrinsic parameters (extrinsic matrix), and the imaging principle of the reference camera (i.e., the projection principle of three-dimensional points onto a two-dimensional plane), each spatial point in the three-dimensional point cloud can be projected onto the imaging screen of each virtual camera to obtain the projection point of each spatial point onto the imaging screen and the pixel position of the projection point in the imaging screen (i.e., pixel coordinates). Then, the sum of the distances between the pixel position of the projection point of each spatial point in the three-dimensional point cloud onto the imaging screen of each virtual camera and the pixel position of the center of the imaging screen of each virtual camera can be calculated to obtain the spatial point loss. Among them, the spatial point loss L can be expressed as formula (1), for example:

[0063]

[0064] Among them, X t,k represents the pixel position of the projection point in the imaging image of the kth virtual camera projected from the tth spatial point, X O,k represents the pixel position of the center of the imaging screen of the kth virtual camera, d(X t,k ,XO,k ) represents X t,k With X o,k , T represents the total number of spatial points in the 3D point cloud, and K represents the total number of generated virtual cameras (that is, the number of cameras).

[0065] Among them, those skilled in the art can use distance calculation methods known in the art, such as Euclidean distance, to calculate the distance between two pixel positions (that is, the distance between the projection point and the center of the imaging picture), and this embodiment of the present disclosure does not limit this.

[0066] As described above, the orientation of the virtual camera can be expressed as the direction vector of the virtual camera. Thus, in each round of optimization, the similarity between the direction vector of each virtual camera and the direction vectors of the remaining virtual cameras can be calculated first, that is, the similarity between the orientations of the virtual cameras is calculated pairwise, and then the similarity is normalized, and then the mean of the normalized similarity (that is, the normalized mean of the similarity) is calculated to obtain the camera diversity loss. Among them, the similarity between two direction vectors can be, for example, cosine similarity. Of course, other similarity algorithms can also be used, and this embodiment of the present disclosure is not limited to this. For example, assuming that there are 10 virtual cameras, the cosine similarity between the orientations (direction vectors) of the 10 virtual cameras is calculated pairwise, and 45 cosine similarities can be obtained. Then, the 45 cosine similarities can be normalized, and then the mean of the 45 normalized cosine similarities is calculated to obtain the camera diversity loss. Among them, those skilled in the art can adopt any normalization method known in the art, for example, dividing the 45 cosine similarities by the maximum cosine similarity among the 45 cosine similarities or dividing them by the average of the 45 cosine similarities to obtain 45 normalized cosine similarities, which is not limited in the embodiments of the present disclosure.

[0067] Among them, those skilled in the art can adopt any optimization algorithm known in the art, such as a gradient descent algorithm, an optimization algorithm, etc., to optimize the i-1th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss, and obtain the i-th generation orientation of each virtual camera. This is not limited to the embodiments of the present disclosure.

[0068] It should be understood that if the i-th generation spatial point loss and the i-th generation camera diversity loss have not yet converged (including any one of the i-th generation spatial point loss and the i-th generation camera diversity loss has not converged), then the i+1-th optimization process can be continued based on the i-th generation orientation of each virtual camera with reference to the above-mentioned i-th optimization process until the spatial point loss and the camera diversity loss have converged, and the orientation of each virtual camera optimized when the spatial point loss and the camera diversity loss have converged is determined as the target orientation of each virtual camera. The convergence condition can be selected as needed. For example, when the difference between the values ​​of two or more adjacent spatial point losses is less than a threshold, the spatial point loss is considered to have converged, and when the difference between the values ​​of two or more adjacent camera diversity losses is less than a threshold, the camera diversity loss is considered to have converged.

[0069] In step S14 , a multi-camera layout solution for the target site is determined based on the target orientation of each virtual camera in the multiple virtual cameras.

[0070] The multi-camera layout scheme of the target site may include the orientation and position of each real camera corresponding to each virtual camera; wherein the orientation of each real camera is the target orientation of each corresponding virtual camera, and the position of each real camera includes the position of each virtual camera evenly distributed on the specified outer contour of the three-dimensional point cloud. For example, Figure 3 A schematic diagram of a multi-camera layout scheme is shown, such as Figure 3 As shown, the black rectangle represents the real camera (that is, the position of the real camera in the target scene), and the dotted line starting from the black rectangle represents the direction of the camera, that is, the direction of the camera can be visualized as the direction from the camera position to the camera target point (such as Figure 3 The black dots in the figure are connected by a line.

[0071] According to the multi-camera layout scheme generation method of the embodiment of the present disclosure, by constructing a three-dimensional point cloud of the target site based on the size information of the target site, and generating multiple virtual cameras with a certain number of cameras at preset positions of the three-dimensional point cloud, and then iteratively optimizing the orientation of the virtual cameras based on spatial point loss and camera diversity loss, a multi-camera layout scheme with no camera blind spots and high camera utilization can be automatically and efficiently generated for the target site, that is, the orientations and positions of multiple real cameras to be arranged in the target site are automatically generated. After the cameras are laid out using the generated multi-camera layout scheme, the multiple real cameras can be made to have no blind spots when performing application scenarios such as spatial positioning or target tracking, and have high camera utilization.

[0072] Considering that the above Figure 1The multi-camera layout solution generation method shown is a solution generated under a certain number of cameras. The number of cameras set by the user may not be the optimal number. It is possible to achieve the goal of no blind spots with a smaller number of cameras. Therefore, the embodiment of the present disclosure also provides Figure 4 A flowchart of a multi-camera layout solution generation method is shown in FIG. Figure 4 As shown, the method includes: steps S41 to S43.

[0073] In step S41 , the size information of the target site where the cameras are to be arranged is obtained, and a three-dimensional point cloud of the target site is generated based on the size information. The implementation of step S41 may refer to the above-mentioned step S11 and will not be described in detail here.

[0074] In step S42, the maximum number M of cameras to be arranged and the camera intrinsic parameters are obtained. The maximum number of cameras can be customized according to actual conditions and is not limited in the embodiment of the present disclosure. The following steps S421 to S422 are iteratively executed in a manner of decreasing from the maximum number M of cameras one by one (i.e., decreasing M by one each time):

[0075] Step S421: In the jth generation process, based on the number of cameras N and the corresponding camera intrinsic parameters, N virtual cameras are generated at preset positions of the 3D point cloud; where j ≥ 0 and N = Mj. For example, assuming that the maximum number of cameras M is 29, the 0th generation process generates 29 virtual cameras, the first generation process generates 28 virtual cameras, the second generation process generates 27 virtual cameras, and so on.

[0076] Step S422, iteratively optimize the orientation of each virtual camera in the N virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the N virtual cameras; wherein the target loss function represents the spatial point loss and the camera diversity loss, the spatial point loss represents the sum of the degrees of deviation of each spatial point in the three-dimensional point cloud from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the N virtual cameras and the orientations of the remaining virtual cameras.

[0077] The implementation of steps S421 to S422 can be specifically referred to in the implementation of steps S12 to S13, and will not be repeated here. In step S422, the orientation of each virtual camera in the N virtual cameras is iteratively optimized based on the target loss function to obtain the target orientation of each virtual camera in the N virtual cameras, including:

[0078] The following optimization process is iteratively performed based on the target loss function:

[0079] During the i-th optimization process, determining the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging image of each virtual camera based on the i-1-th generation orientation of each virtual camera obtained during the i-1-th optimization process, and determining the similarity between the i-1-th generation orientation of each virtual camera in the N virtual cameras and the i-1-th generation orientation of the remaining virtual cameras; wherein i ≥ 1, and the generation 0 orientation is a preset initial orientation;

[0080] Determine the i-th generation spatial point loss based on the sum of the distances between the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging screen of each virtual camera and the pixel position of the center of the imaging screen of each virtual camera;

[0081] Determine the diversity loss of the i-th generation camera according to the normalized mean of the similarities between the i-1-th generation orientation of each virtual camera in the N virtual cameras and the i-1-th generation orientation of the remaining virtual cameras;

[0082] When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, optimizing the i-1-th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera;

[0083] When both the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

[0084] In step S43, when the loss value suddenly increases when the target loss function converges, the multi-camera layout scheme when the number of cameras is N+1 is determined as the target multi-camera layout scheme for the target site. The multi-camera layout scheme is determined according to the target orientation of each virtual camera in the N+1 virtual cameras, and the loss value when the target loss function converges includes the loss value when the target orientation of each virtual camera in the N virtual cameras is determined.

[0085] The loss value when the target loss function converges can be understood as the loss value of the spatial point loss and the camera diversity loss calculated when the target orientations of the virtual cameras with the number of N are obtained, that is, the loss value when the spatial point loss and the camera diversity loss converge. If the loss value when the target loss function corresponding to the number of N cameras converges (that is, the spatial point loss and the camera diversity loss) suddenly increases, it can be considered that the loss value when the target loss function corresponding to the number of N cameras converges obviously increases, and it is represented that the target site cannot achieve the goal of no blind area by using N real cameras, that is, it means that the target site needs to layout at least N+1 real cameras, and N+1 real cameras are the minimum number of cameras that the target site needs to layout. Therefore, the multi-camera layout scheme when the number of cameras is N+1 can be determined based on the target orientations of the virtual cameras in the N+1 virtual cameras, and the multi-camera layout scheme when the number of cameras is N+1 is determined as the target multi-camera layout scheme of the target site.

[0086] The loss value when the target loss function converges can be understood as the loss value of the spatial point loss and the camera diversity loss calculated when the target orientations of the virtual cameras with the number of N are obtained, that is, the loss value when the spatial point loss and the camera diversity loss converge. If the loss value when the target loss function corresponding to the number of N cameras converges (that is, the spatial point loss and the camera diversity loss) suddenly increases, it can be considered that the loss value when the target loss function corresponding to the number of N cameras converges obviously increases, and it is represented that the target site cannot achieve the goal of no blind area by using N real cameras, that is, it means that the target site needs to layout at least N+1 real cameras, and N+1 real cameras are the minimum number of cameras that the target site needs to layout. Therefore, the multi-camera layout scheme when the number of cameras is N+1 can be determined based on the target orientations of the virtual cameras in the N+1 virtual cameras, and the multi-camera layout scheme when the number of cameras is N+1 is determined as the target multi-camera layout scheme of the target site.

[0087] Among them, the multi-camera layout scheme when the number of cameras is N+1 includes the orientation and position of each real camera in the N+1 real cameras corresponding to the N+1 virtual cameras; wherein, the orientation of each real camera is the target orientation of the corresponding virtual camera, and the position of each real camera corresponds to the preset position; optionally, the preset position includes the uniformly distributed position on the specified outer contour of the three-dimensional point cloud, that is, the position of each real camera includes the uniformly distributed position of each virtual camera on the specified outer contour of the three-dimensional point cloud, and the specified outer contour includes the outer contour of the highest height of the three-dimensional point cloud.

[0088] The multi-camera layout solution generation method implemented in steps S41 to S43 can be understood as iteratively repeating steps S421 to S422, with the number of cameras reduced by one at each iteration. The iteration is terminated when the loss value corresponding to a particular generation of multi-camera layout solutions increases significantly when the target loss function converges. The number of cameras at this point is denoted as N, and the multi-camera layout solution with N+1 cameras is adopted as the target multi-camera layout solution for the target site. For example, if the maximum number of cameras is 29, and the loss value at convergence of the target loss function corresponding to 26 cameras in the third generation process is greater than a preset threshold, the multi-camera layout solution generated in the second generation process with 27 cameras can be determined as the target multi-camera layout solution for the target site. It should be understood that if the loss value when the target loss function converges when the number of cameras corresponding to the third generation process is 26 is still less than or equal to the preset threshold, the generation process of the above steps S421 to S422 can be continued based on the number of cameras being 25 until the loss value when the target loss function converges increases suddenly, so as to obtain the target multi-camera layout plan for the target site.

[0089] According to the multi-camera layout solution generation method of the embodiment of the present disclosure, by generating multi-camera layout solutions with different camera numbers in an iterative manner that decreases one by one from the maximum number of cameras M, it is possible to automatically and efficiently determine the minimum number of cameras required to eliminate camera blind spots in the target site and the orientation and position of each real camera with the minimum number of cameras, while achieving a high camera utilization rate.

[0090] The multi-camera layout scheme generation method according to the embodiment of the present disclosure can solve the target multi-camera layout scheme adapted to the current target site through optimization based on the external input target site size information. Compared with manually designing the layout scheme based on experience, the embodiment of the present disclosure directly uses the spatial point loss and camera diversity loss of the target site as optimization targets, which can ensure that the final target multi-layout scheme has no tracking blind spots and the highest camera utilization rate. The entire process can be completed automatically without human intervention, and the design and generation of the target multi-layout scheme can be completed in seconds, which is more efficient.

[0091] According to the multi-camera layout solution generation method of the embodiment of the present disclosure, an optimization algorithm is used to solve a target multi-camera layout solution that is adapted to the target site. The solution includes the number of cameras and the pose (i.e., position and orientation) of each camera, and the target site is defined and represented by a three-dimensional point cloud. The target loss function is used to quantify the site tracking blind spots and camera utilization, and optimization iterations are performed based on this goal. This method can better ensure the stability and availability of the multi-camera layout solution.

[0092] based on Figure 1 The generation method shown is Figure 5 A block diagram of a multi-camera layout solution generating device according to an embodiment of the present disclosure is shown as follows: Figure 5 As shown, the device includes:

[0093] The point cloud generation module 501 is used to obtain the size information of the target site where the camera is to be arranged, and generate a three-dimensional point cloud of the target site based on the size information;

[0094] A virtual camera generation module 502 is configured to obtain the number of cameras to be arranged and camera intrinsic parameters, and generate a plurality of virtual cameras at preset positions of the three-dimensional point cloud based on the number of cameras and the camera intrinsic parameters;

[0095] an orientation optimization module 503 for iteratively optimizing the orientation of each virtual camera in the plurality of virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the plurality of virtual cameras; wherein the target loss function represents a spatial point loss and a camera diversity loss, wherein the spatial point loss represents the sum of the degrees to which each spatial point in the three-dimensional point cloud deviates from the center of the imaging image after being projected onto the imaging image of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the plurality of virtual cameras and the orientations of the remaining virtual cameras;

[0096] The solution generating module 504 is configured to determine a multi-camera layout solution for the target site based on the target orientation of each virtual camera in the multiple virtual cameras.

[0097] In a possible implementation, the iteratively optimizing the orientation of each virtual camera in the multiple virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the multiple virtual cameras includes:

[0098] The following optimization process is iteratively performed based on the objective loss function:

[0099] During the i-th optimization process, determining the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging image of each virtual camera based on the i-1-th generation orientation of each virtual camera obtained during the i-1-th optimization process, and determining the cosine similarity between the i-1-th generation orientation of each virtual camera in the plurality of virtual cameras and the i-1-th generation orientations of the remaining virtual cameras; wherein i ≥ 1, and the generation 0 orientation is a preset initial orientation;

[0100] Determine the i-th generation spatial point loss based on the sum of the distances between the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging screen of each virtual camera and the pixel position of the center of the imaging screen of each virtual camera;

[0101] determining an i-th generation camera diversity loss according to a normalized mean of cosine similarities between an i-1-th generation orientation of each virtual camera in the plurality of virtual cameras and an i-1-th generation orientation of the remaining virtual cameras;

[0102] When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, optimizing the i-1-th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera;

[0103] When the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

[0104] In one possible implementation, the multi-camera layout scheme of the target site includes the orientation and position of each real camera corresponding to each virtual camera; wherein the orientation of each real camera is the target orientation of the corresponding virtual camera, and the position of each real camera corresponds to the preset position, and the preset position includes a uniformly distributed position on the specified outer contour of the three-dimensional point cloud, and the specified outer contour includes the outer contour of the highest height of the three-dimensional point cloud.

[0105] According to the device of the embodiment of the present disclosure, by constructing a three-dimensional point cloud of the target site based on the size information of the target site, generating multiple virtual cameras with a certain number of cameras at preset positions of the three-dimensional point cloud, and then iteratively optimizing the orientation of the virtual cameras based on spatial point loss and camera diversity loss, it is possible to automatically and efficiently generate a multi-camera layout solution for the target site, that is, automatically generate the orientations and positions of multiple real cameras to be arranged in the target site, and after using the generated multi-camera layout solution to perform camera layout, multiple real cameras can be made to have no tracking blind spots when performing application scenarios such as spatial positioning or target tracking, and have a high camera utilization rate.

[0106] based on Figure 4 The generation method shown is Figure 6 A block diagram of a multi-camera layout solution generating device according to an embodiment of the present disclosure is shown as follows: Figure 6 As shown, the device includes:

[0107] A three-dimensional point cloud generation module 601 is used to obtain size information of a target site where cameras are to be arranged, and generate a three-dimensional point cloud of the target site based on the size information;

[0108] The iterative generation module 602 is used to obtain the maximum number M of cameras to be arranged and the camera intrinsic parameters, and iteratively perform the following generation process in a decreasing manner starting from the maximum number M of cameras:

[0109] In the jth generation process, based on the number of cameras N and the corresponding camera intrinsic parameters, N virtual cameras are generated at the preset positions of the three-dimensional point cloud; wherein j≥0, N=Mj;

[0110] Iteratively optimizing the orientation of each virtual camera in the N virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the N virtual cameras; wherein the target loss function represents a spatial point loss and a camera diversity loss, the spatial point loss represents the sum of the degrees to which each spatial point in the three-dimensional point cloud deviates from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the N virtual cameras and the orientations of the remaining virtual cameras;

[0111] The layout scheme determination module 603 is used to determine the multi-camera layout scheme when the number of cameras is N+1 as the target multi-camera layout scheme of the target site when the loss value suddenly increases when the target loss function converges. The loss value when the target loss function converges includes the loss value when the target orientation of each virtual camera in the N virtual cameras is determined.

[0112] In one possible implementation, iteratively optimizing the orientation of each virtual camera in the N virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the N virtual cameras includes iteratively performing the following optimization process based on the target loss function:

[0113] During the i-th optimization process, determining the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging image of each virtual camera based on the i-1-th generation orientation of each virtual camera obtained during the i-1-th optimization process, and determining the cosine similarity between the i-1-th generation orientation of each virtual camera in the N virtual cameras and the i-1-th generation orientation of the remaining virtual cameras; wherein i ≥ 1, and the generation 0 orientation is a preset initial orientation;

[0114] Determine the i-th generation spatial point loss based on the sum of the distances between the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging screen of each virtual camera and the pixel position of the center of the imaging screen of each virtual camera;

[0115] Determining the i-th generation camera diversity loss according to a normalized mean of cosine similarities between the i-1th generation orientation of each virtual camera in the N virtual cameras and the i-1th generation orientations of the remaining virtual cameras;

[0116] When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, optimizing the i-1-th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera;

[0117] When the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

[0118] In one possible implementation, the multi-camera layout scheme when the number of cameras is N+1 includes the orientation and position of each real camera among the N+1 real cameras corresponding to the N+1 virtual cameras; wherein the orientation of each real camera is the target orientation of the corresponding virtual camera, and the position of each real camera corresponds to the preset position, and the preset position includes a uniformly distributed position on the specified outer contour of the three-dimensional point cloud, and the specified outer contour includes the outer contour of the highest height of the three-dimensional point cloud.

[0119] According to the apparatus of the embodiment of the present disclosure, by generating multi-camera layout solutions under various camera numbers in a manner that decreases one by one from the maximum number of cameras M, it is possible to automatically and efficiently determine the minimum number of cameras required to eliminate tracking blind spots in the target site and the orientation and position of each real camera under the minimum number of cameras, and achieve high camera utilization.

[0120] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0121] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0122] The embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0123] The embodiment of the present disclosure further provides a computer program product, comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in the processor of the electronic device, the processor in the electronic device executes the above method.

[0124] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 can be provided as a server or a terminal device. Referring to Figure 7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application program. The application program stored in the memory 1932 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above method.

[0125] The electronic device 1900 can also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0126] In an exemplary embodiment, a non-volatile computer readable storage medium, such as the memory 1932 including computer program instructions executable by the processing component 1922 of the electronic device 1900 to complete the above method, is also provided.

[0127] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0128] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0129] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0130] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0131] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0132] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0133] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0134] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0135] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a multi-camera layout solution, characterized in that: include: Acquire size information of a target site where cameras are to be arranged, and generate a three-dimensional point cloud of the target site based on the size information; Obtaining the number of cameras to be arranged and camera intrinsic parameters, and generating a plurality of virtual cameras at preset positions of the three-dimensional point cloud based on the number of cameras and the camera intrinsic parameters; wherein the preset positions include positions evenly distributed on a specified outer contour of the three-dimensional point cloud; Iteratively optimizing the orientation of each virtual camera in the multiple virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the multiple virtual cameras; wherein the target loss function represents a spatial point loss and a camera diversity loss, the spatial point loss represents the sum of the degrees to which each spatial point in the three-dimensional point cloud deviates from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the multiple virtual cameras and the orientations of the remaining virtual cameras; Based on the target orientation of each virtual camera among the multiple virtual cameras, a multi-camera layout scheme for the target site is determined, wherein the multi-camera layout scheme for the target site includes the orientation and position of each real camera corresponding to each virtual camera; wherein the orientation of each real camera is the target orientation of the corresponding virtual camera, and the position of each real camera includes the position of each virtual camera evenly distributed on a specified outer contour of the three-dimensional point cloud.

2. The method according to claim 1, characterized in that The iteratively optimizing the orientation of each virtual camera in the multiple virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the multiple virtual cameras includes: The following optimization process is iteratively performed based on the objective loss function: During the i-th optimization process, determining the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging image of each virtual camera based on the i-1-th generation orientation of each virtual camera obtained during the i-1-th optimization process, and determining the similarity between the i-1-th generation orientation of each virtual camera in the plurality of virtual cameras and the i-1-th generation orientations of the remaining virtual cameras; wherein i ≥ 1, and the generation 0 orientation is a preset initial orientation; Determine the i-th generation spatial point loss based on the sum of the distances between the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging screen of each virtual camera and the pixel position of the center of the imaging screen of each virtual camera; determining an i-th generation camera diversity loss according to a normalized mean of similarities between an i-1-th generation orientation of each virtual camera in the plurality of virtual cameras and an i-1-th generation orientation of the remaining virtual cameras; When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, optimizing the i-1-th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera; When the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

3. The method according to claim 1 or 2, characterized in that The multi-camera layout scheme of the target site includes the orientation and position of each real camera corresponding to each virtual camera; The orientation of each real camera is the target orientation of each corresponding virtual camera, and the position of each real camera corresponds to the preset position.

4. The method according to claim 1 or 2, characterized in that The preset positions include positions evenly distributed on a specified outer contour of the three-dimensional point cloud.

5. The method according to claim 4, characterized in that The designated outer contour includes the outer contour of the three-dimensional point cloud at the highest height.

6. A method for generating a multi-camera layout solution, characterized in that: include: Acquire size information of a target site where cameras are to be arranged, and generate a three-dimensional point cloud of the target site based on the size information; Obtain the maximum number of cameras M to be laid out and the camera intrinsic parameters, and iteratively execute the following generation process in a descending order starting from the maximum number of cameras M: In the jth generation process, based on the number of cameras N and the corresponding camera intrinsic parameters, N virtual cameras are generated at the preset positions of the three-dimensional point cloud; wherein j≥0, N=Mj; Iteratively optimizing the orientation of each virtual camera in the N virtual cameras based on a target loss function to obtain a target orientation of each virtual camera in the N virtual cameras; wherein the target loss function represents a spatial point loss and a camera diversity loss, the spatial point loss represents the sum of the degrees to which each spatial point in the three-dimensional point cloud deviates from the center of the imaging screen after being projected onto the imaging screen of each virtual camera, and the camera diversity loss represents the similarity between the orientation of any virtual camera in the N virtual cameras and the orientations of the remaining virtual cameras; In the case that the loss value suddenly increases when the target loss function converges, the multi-camera layout scheme when the number of cameras is N+1 is determined as the target multi-camera layout scheme for the target site. The multi-camera layout scheme is determined according to the target orientation of each virtual camera in the N+1 virtual cameras. The loss value when the target loss function converges includes the loss value when the target orientation of each virtual camera in the N virtual cameras is determined.

7. The method according to claim 6, characterized in that The iteratively optimizing the orientation of each virtual camera in the N virtual cameras based on the target loss function to obtain the target orientation of each virtual camera in the N virtual cameras includes: The following optimization process is iteratively performed based on the objective loss function: During the i-th optimization process, determining the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging image of each virtual camera based on the i-1-th generation orientation of each virtual camera obtained during the i-1-th optimization process, and determining the similarity between the i-1-th generation orientation of each virtual camera in the N virtual cameras and the i-1-th generation orientation of the remaining virtual cameras; wherein i ≥ 1, and the generation 0 orientation is a preset initial orientation; Determine the i-th generation spatial point loss based on the sum of the distances between the pixel position of each spatial point in the three-dimensional point cloud projected onto the imaging screen of each virtual camera and the pixel position of the center of the imaging screen of each virtual camera; Determine the i-th generation camera diversity loss according to a normalized mean of similarities between the i-1-th generation orientation of each virtual camera in the N virtual cameras and the i-1-th generation orientations of the remaining virtual cameras; When the i-th generation spatial point loss and the i-th generation camera diversity loss have not converged, optimizing the i-1-th generation orientation of each virtual camera based on the i-th generation spatial point loss and the i-th generation camera diversity loss to obtain the i-th generation orientation of each virtual camera; When the i-th generation spatial point loss and the i-th generation camera diversity loss have converged, the i-th generation orientation of each virtual camera is determined as the target orientation of each virtual camera.

8. The method according to claim 6 or 7, characterized in that The multi-camera layout solution when the number of cameras is N+1 includes the orientation and position of each real camera in the N+1 real cameras corresponding to the N+1 virtual cameras; The orientation of each real camera is the target orientation of each corresponding virtual camera, and the position of each real camera corresponds to the preset position.

9. The method according to claim 6 or 7, characterized in that The preset positions include positions evenly distributed on a specified outer contour of the three-dimensional point cloud.

10. The method according to claim 9, characterized in that The designated outer contour includes the outer contour of the three-dimensional point cloud at the highest height.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 10 when executing the instructions stored in the memory.

12. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

13. A computer program product comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code, wherein when the computer-readable code is executed in a processor of an electronic device, the processor in the electronic device executes the method according to any one of claims 1 to 10.

Images