A spatial positioning method and device, storage medium and electronic device

Abstract

The present application discloses a spatial positioning method and device, a storage medium and an electronic device, the method comprising: determining target image data corresponding to the acquisition device according to removing detection objects in image data acquired by an acquisition device in a positioning scene; inputting the target image data and depth map data of the target image data into a neural network in the form of data pairs, obtaining point cloud data of the target image data in a physical world coordinate system, and posture data of the acquisition device corresponding to the target image data in the physical world coordinate system; determining the positioning information of the target object in the target image data in the physical world coordinate system according to the ground plane and posture data in the target image data detected by the point cloud data; thereby achieving precise positioning without being limited to the installation angle, quantity and other restrictions of the acquisition device, and being able to achieve spatial positioning through any acquisition device, thereby reducing costs.

Description

Technical Field

[0001] The present application relates to computer technology, and more particularly to a method and device for spatial positioning. The present application also relates to a method and device for constructing a thermal map, as well as a computer storage medium and an electronic device. Background Art

[0002] In the passenger flow scene, efficient and accurate spatial positioning of objects in the camera screen is a prerequisite for helping merchants to carry out refined operations and provide personalized services. For example, in shopping malls and 4S stores, the pixel positions in the captured images can be converted into real physical world coordinates through calibrated cameras, thereby realizing passenger flow analysis. In shopping malls, by converting the detected customer coordinates into real physical coordinates, the ambiguity of distance in two-dimensional images can be effectively avoided, and more accurate position analysis can be achieved; in 4S stores, by converting the detected display car pixel coordinates into real physical coordinates, it can effectively cooperate with passenger flow analysis and understand the attention of different vehicles. Therefore, in order to provide more accurate passenger flow analysis services, it is necessary to efficiently and accurately position the objects in the captured images. Summary of the invention

[0003] The present application provides a spatial positioning method to solve the problems of poor positioning accuracy and high cost in the prior art.

[0004] The present application provides a spatial positioning method, comprising:

[0005] Determine target image data corresponding to the acquisition device by removing the detection object from the image data acquired by the acquisition device in the positioning scene;

[0006] Inputting the target image data and the depth map data of the target image data into a neural network in the form of data pairs, obtaining point cloud data of the target image data in a physical world coordinate system, and position and posture data of the acquisition device corresponding to the target image data in the physical world coordinate system;

[0007] According to the ground plane and the posture data in the target image data detected by the point cloud data, the positioning information of the target object in the target image data in the physical world coordinate system is determined.

[0008] In some embodiments, the step of removing the detection object from the image data acquired by the acquisition device in the positioning scene to determine the target image data corresponding to the acquisition device includes:

[0009] Acquiring the image and video data collected by the acquisition device in the positioning scene according to the set acquisition time;

[0010] Detecting a plurality of image frames in the image video data to determine whether the detection object exists in the plurality of image frames;

[0011] If so, remove the detection object in the detection frame;

[0012] According to the multiple image frames after the removal, target image data corresponding to the acquisition device is determined.

[0013] In some embodiments, removing the detection object comprises:

[0014] Determine whether a hole area exists in the multiple image frames according to the acquired pixel information of the multiple image frames;

[0015] If yes, then fill the hole area;

[0016] The step of determining target image data corresponding to the acquisition device according to the removed multiple image frames includes:

[0017] According to the completed image frame, the target image data corresponding to the acquisition device is determined.

[0018] In some embodiments, determining whether a hole area exists in the multiple image frames according to the acquired pixel information of the multiple image frames includes:

[0019] Counting the number of times that pixels at the same position of the multiple image frames are located outside the detection frame;

[0020] Determine the mean value of pixels at the same position according to the number of occurrences;

[0021] Combining the multiple image frames into a composite frame image according to the pixel mean value;

[0022] Determine whether there is a detection frame area with empty pixel values ​​in the synthesized frame image, and if so, determine that there is a hole area in the synthesized frame image.

[0023] In some embodiments, the filling of the hole region includes:

[0024] Acquire areas corresponding to the detection frames in the multiple image frames to determine a set of detection frame areas;

[0025] According to the set area selection requirement, a target detection frame that meets the selection requirement is selected from the detection frame area set;

[0026] Completing the hole area in the synthesized frame image according to pixels at corresponding positions in the image frame where the target detection frame is located;

[0027] The step of determining the target image data corresponding to the acquisition device according to the completed image frame includes:

[0028] The completed synthetic frame image is determined as the target image data corresponding to the acquisition device.

[0029] In some embodiments, the inputting the target image data and the depth map data of the target image data into the neural network in the form of data pairs, acquiring the point cloud data of the target image data in the world coordinate system, and the pose data of the acquisition device corresponding to the target image data in the world coordinate system, includes:

[0030] Combining the target image data into the image pair in the form of data pairs;

[0031] composing a depth map pair according to the acquired depth map of the target image data in the image pair;

[0032] Inputting the target image data in the image pair and the depth map corresponding to the target image data in the depth map pair as an input data pair into the neural network, and acquiring local point cloud data of the input data pair and parameters of the acquisition device corresponding to the input data pair;

[0033] The pose data of the acquisition device in the world coordinate system and the global point cloud data of the target image data in the world coordinate system are determined according to the parameters and the local point cloud data.

[0034] In some embodiments, determining the positioning information of the target object in the target image data in the physical world coordinate system based on the ground plane and the pose data in the target image data detected by the point cloud data includes:

[0035] Determining a predicted point cloud plane according to a point cloud reference point selected from the point cloud data;

[0036] Determine the distance from the point cloud reference point to the predicted point cloud plane, and count the distances between the point cloud reference point and the predicted point cloud plane. The number of reference points of the point cloud within the range;

[0037] Determine whether the number of the point cloud reference points is greater than a set threshold, and if so, determine the predicted point cloud plane as the point cloud reference plane;

[0038] Iteratively executing the step of determining the predicted point cloud plane, when the number of point cloud reference points in the point cloud reference plane after the iteration is greater than the number of point cloud reference points in the point cloud reference plane before the iteration, updating the point cloud reference plane until the number of iterations meets the number requirement and stops the iteration;

[0039] According to the updated point cloud reference plane and the position and posture data, the positioning information of the target object in the target image data in the world coordinate system is determined.

[0040] In some embodiments, determining the positioning information of the target object in the target image data in the world coordinate system according to the updated point cloud reference plane and the pose data includes:

[0041] A point cloud reference plane that is rotated and / or translated to be in the same plane as the positioning scene map is determined as the ground plane in the physical world coordinate system;

[0042] According to the rotation and / or translation of the point cloud reference plane, performing the same rotation and / or translation on the acquisition device;

[0043] According to the rotation and / or translation of the acquisition device, updating the position and posture data of the acquisition device in the physical world coordinate system;

[0044] According to the ground plane in the physical world coordinate system and the updated posture data, the positioning information of the target object in the target image data in the world coordinate system is determined.

[0045] In some embodiments, the step of rotating and / or translating the point cloud reference plane that is in the same plane as the positioning scene map to determine the ground plane in the physical world coordinate system includes:

[0046] Determining a rotation axis according to a normal vector of the point cloud reference plane and a normal vector of the positioning scene map;

[0047] Determine the cosine value and sine value of the angle between the normal vector of the point cloud reference plane and the normal vector of the positioning scene map according to the rotation axis;

[0048] Determine a rotation matrix according to the rotation axis, the cosine value and the sine value;

[0049] Rotate the point cloud data and the point cloud reference plane according to the rotation matrix to determine a rotated point cloud reference plane;

[0050] The rotated point cloud reference plane is translated to a translated point cloud reference plane that is located at the same plane position as the positioning scene map, and is determined as the ground plane in the physical world coordinate system.

[0051] In some embodiments, the step of translating the rotated point cloud reference plane to a position where the translated point cloud reference plane is located at the same plane position as the positioning scene map, and determining the translated point cloud reference plane as the ground plane in the physical world coordinate system, comprises:

[0052] Matching the translated point cloud reference plane with the positioning scene map;

[0053] The matched point cloud reference plane is determined as the ground plane in the physical world coordinate system.

[0054] In some embodiments, the step of determining a predicted point cloud plane based on selecting a reference point of a point cloud in the point cloud data includes:

[0055] Randomly selecting a point cloud reference point in the point cloud data;

[0056] The point cloud reference points are fitted into a predicted point cloud plane.

[0057] In some embodiments, determining the positioning information of the target object in the target image data in the world coordinate system based on the ground plane and the posture data in the target image data detected by the point cloud data includes:

[0058] Determining, according to the position and posture data, a ray extending from the acquisition device to the target object located in the ground plane;

[0059] The position information of the intersection point between the ray and the ground plane in the physical world coordinate system is determined as the positioning information of the target object in the physical world coordinate system.

[0060] The present application also provides a spatial positioning device, comprising:

[0061] A first determining unit, configured to determine target image data corresponding to the acquisition device by removing the detection object from the image data acquired by the acquisition device in the positioning scene;

[0062] An acquisition unit, used to input the target image data and the depth map data of the target image data into a neural network in the form of data pairs, and acquire point cloud data of the target image data in a physical world coordinate system, and position and posture data of the acquisition device corresponding to the target image data in the physical world coordinate system;

[0063] The second determining unit is used to determine the positioning information of the target object in the target image data in the world coordinate system according to the ground plane and the posture data in the target image data detected by the point cloud data.

[0064] The present application also provides a method for constructing a heat map, including:

[0065] According to the above-mentioned spatial positioning method, the positioning information of the target object in the target image data collected by the acquisition device in the physical world coordinate system is obtained;

[0066] Detecting a moving object within a preset range of the target object according to the positioning information;

[0067] According to the number of moving objects, a heat map of the target object is constructed.

[0068] The present application also provides a computer storage medium, including a computer program. When the computer program is run on an electronic device, the electronic device executes the above-mentioned spatial positioning method, or executes the above-mentioned thermal map construction method.

[0069] The present application also provides an electronic device, including:

[0070] processor;

[0071] The memory is used to store a program for processing data generated by the electronic device. When the program is read and executed by the processor, it executes the above-mentioned spatial positioning method, or the above-mentioned thermal map construction method.

[0072] Compared with the prior art, this application has the following advantages:

[0073] A spatial positioning method provided by the present application is no longer constrained by the installation angle, installation quantity, etc. of acquisition equipment such as cameras, and can realize three-dimensional reconstruction of the positioning space under the condition of sparse field of view. The three-dimensional reconstruction can be based on the obtained point cloud data, and the ground plane and camera posture data in the target image data can be detected through the point cloud data, so as to determine the positioning information of the target object in the physical world coordinate system, which can not only improve the accuracy of positioning but also reduce the cost of positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a flow chart of a spatial positioning method provided by this application.

[0075] Figure 2 It is a flow chart of an embodiment of detecting object removal in a spatial positioning method provided in the present application.

[0076] Figure 3 It is a structural schematic diagram of a neural network embodiment in a spatial positioning method provided in this application.

[0077] Figure 4 It is a flowchart of an embodiment of determining the positioning information of a target object in physical world coordinates in a spatial positioning method provided by the present application.

[0078] Figure 5 It is a schematic diagram of a positioning embodiment in a spatial positioning method provided in the present application.

[0079] Figure 6It is a structural schematic diagram of a space positioning device provided by this application.

[0080] Figure 7 It is a flow chart of a method for constructing a heat map provided in this application.

[0081] Figure 8 It is a structural schematic diagram of a thermal map construction device provided in this application.

[0082] Fig. 9 It is a structural schematic diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0083] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0084] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The descriptions used in this application and the appended claims, such as "a", "first", and "second", are not limitations on quantity or sequence, but are used to distinguish the same type of information from each other.

[0085] According to the above background technology, it can be known that the invention concept of this application is derived from the need for passenger flow analysis. Of course, it is only the motivation for the invention in this scenario, which does not mean that the technical solution provided is limited to this scenario. It is applicable to other application scenarios outside this scenario if it involves spatial positioning, such as: unmanned driving, navigation, target positioning, etc.

[0086] In the prior art, the commonly used method for spatial positioning is to collect video through a binocular camera or a pan-tilt camera, and make the target be located in the center of the video image, and then use the position information of the control system to locate the target. However, in the data flow scene, the position of the camera is relatively fixed, and adding a control system alone to achieve target positioning will lead to increased costs and poor positioning accuracy. If the control system is not considered to be added alone, the pixel coordinates in the video image can be mapped to the world coordinates by using the camera's internal and external parameters and distortion parameters, but this method requires the camera parameters to be calibrated in advance to obtain the internal and external parameters and distortion parameters. In the data flow scene, the acquisition of camera parameters depends on manual calibration, which will lead to a significant increase in labor costs and affect the positioning accuracy.

[0087] Based on the above, the present application provides a spatial positioning method, which has no constraints on the installation method and installation position of the acquisition equipment in the positioning scene, and can achieve high-precision and low-cost positioning in sparse scenes, data flow scenes, etc. without manual intervention and setting of related reference equipment. A spatial positioning method provided by the present application is described in detail below in combination with a specific implementation method.

[0088] like Figure 1 As shown, Figure 1 This is a flow chart of a spatial positioning method provided by the present application, the method comprising:

[0089] Step S101: Determine target image data corresponding to the acquisition device by removing detection objects from the image data acquired by the acquisition device in the positioning scene;

[0090] Step S102: inputting the target image data and the depth map data of the target image data into the neural network in the form of data pairs, acquiring point cloud data of the target image data in a physical world coordinate system, and position and posture data of the acquisition device corresponding to the target image data in the physical world coordinate system;

[0091] Step S103: determining the positioning information of the target object in the target image data in the physical world coordinate system according to the ground plane and the posture data in the target image data detected by the point cloud data.

[0092] In the embodiment of the present application, the positioning scene can be any scene where a collection device is installed, for example, some public places such as shopping malls, stations, subways, buses, etc., and can also be scenes involved in devices such as cars. The collection device can be an electronic device such as a camera or a webcam. The above steps S101 to S103 are described in detail below in conjunction with a specific implementation method.

[0093] Regarding step S101: according to removing the detection object in the image data acquired by the acquisition device in the positioning scene, the target image data corresponding to the acquisition device is determined.

[0094] The purpose of step S101 is to remove the detection object from the video image data collected by the acquisition device. In this embodiment, the target object to be located may be an object that does not change within a set time range in the image data, such as a car on a booth in a 4S store, a decoration in a store in a shopping mall, etc. Correspondingly, the detection object may be an object that changes in the image data, such as a human body in a 4S store or a shopping mall. The target image data may be an image after the human body is removed from the image data. Therefore, if Figure 2 As shown, Figure 2This is a flowchart of an embodiment of detecting object removal in a spatial positioning method provided by the present application. The specific implementation process of step S101 may include:

[0095] Step S101-11: According to the set acquisition time, the image and video data collected by the acquisition device in the positioning scene are obtained; because in the positioning scene, there are usually multiple acquisition devices, but of course there can also be one. In order to ensure the consistency of the images collected by acquisition devices at different locations, in this embodiment, the acquisition device obtains the image and video data according to the set acquisition time, for example: the video data of the acquisition devices in a shorter time range is obtained, and the shorter time range can be determined according to empirical values, such as obtaining image and video data within a range of 20-40 seconds. Assuming that the positioning scene includes I acquisition devices, the video set can be obtained . For the image video data , Take J frames as image frames, and each image frame is recorded as , .

[0096] Step S101-12: Detect multiple image frames in the image video data to determine whether the detection object exists in the multiple image frames; input the multiple image frames into the detection model, which can be Yolo V5, of course, it can also be a human posture estimation model, Faster R-CNN (target detection model), etc. The detection method used for the detection object in this embodiment is not limited to the above examples, and any method that can be used for target detection is included. The detection box in the image frame can be obtained by detection. , The detection frame is the detection object, for example, a human body. Where N is the number of all detection frames, that is, the total number of detection frames in all frames under the i-viewing angle, and n is the number of the current frame. Human body detection or target detection can be obtained by using the detection model provided in the prior art, which will not be described in detail here.

[0097] Step S101-13: If yes, remove the detection object in the detection frame; the specific implementation process of the removal in this embodiment may include:

[0098] Step S101-131: determining whether a hole area exists in the plurality of image frames according to the acquired pixel information of the plurality of image frames;

[0099] Step S101-132: If yes, then fill the hole area;

[0100] Step S101 - 133 : Determine the target image data corresponding to the acquisition device according to the completed image frame.

[0101] The specific implementation process of steps S101-131 may include:

[0102] Step S101-1311: Count the number of times that the pixels at the same position of the multiple image frames are outside the detection frame; for example: traverse the pixel values ​​of the image frames ,in . Assume that the set of detection boxes in the image frame is Identify pixels at the same position in multiple image frames at the same viewing angle, and count the number of pixels in the multiple image frames that are not in the detection frame. The number of times s appears in the image frame. The multiple image frames at the same viewing angle may be multiple image frames of the video captured at the same viewing angle when multiple cameras (capture devices) are provided in the positioning scene. When one camera is provided in the positioning scene, it refers to multiple image frames in the video captured by the camera at the same viewing angle. It is understandable that when there are multiple cameras, the multiple image frames of the video captured at the same viewing angle may be multiple image frames of the multiple cameras at the same viewing angle determined by selecting some of the multiple cameras in combination with the positioning requirements.

[0103] Step S101-1312: Determine the mean value of pixels at the same position according to the number of occurrences; the calculation of the mean value of pixels in this embodiment may be performed using the following formula 1:

[0104]

[0105] Formula 1

[0106] in, is a sign function, which takes 1 when the condition is met, otherwise it takes 0. Based on the above formula 1, the pixel mean can be obtained .

[0107] Step S101-1313: According to the pixel mean, the multiple image frames are synthesized into a synthesized frame image; in this embodiment, the pixel mean can be obtained according to the above formula 1 , according to the pixel mean, the synthetic frame image can be obtained .

[0108] Step S101-1314: Determine whether there is a detection frame area with empty pixel values ​​in the synthesized frame image. If so, determine whether there is a hole area in the synthesized frame image. The hole area can be represented as , then, the synthetic frame image The hole area in can be expressed as shown in Formula 2:

[0109] Formula 2

[0110] In this embodiment, if there are empty pixels in the synthetic frame image, such as zero, it can be determined that there is a hole area in the detection frame area. That is to say, the position of the detection object has not changed or has moved within a smaller range, resulting in the existence of a hole area in the synthetic frame image. Therefore, the specific implementation process of filling the hole area in step S101-132 may include:

[0111] Step S101-1321: Obtain the areas corresponding to the detection frames in the multiple image frames and determine a set of detection frame areas. In this embodiment, assume that the detection frame The corresponding area is According to the areas corresponding to the detection frames of the multiple image frames, a set of detection frame areas can be obtained.

[0112] Step S101-1322: According to the set area selection requirement, select a target detection frame that meets the selection requirement from the detection frame area set; in this embodiment, the detection frame with the smallest area in the detection frame area set can be selected as the target detection frame , as shown in Formula 3:

[0113] Formula 3

[0114] Step S101-1323: Fill the hole area in the synthesized frame image according to the pixels at the corresponding position in the image frame where the target detection frame is located. In this embodiment, the pixel value at the same position in the minimum detection frame can be assigned to the synthesized frame image. At the same position of , the following formula 4 can be used to achieve (where, is the minimum detection box):

[0115] Formula 4

[0116] The above formula can be understood as: if the pixel position is Within the range of The pixel value corresponding to the pixel position in is assigned to the pixel at the same position in the synthesized frame.

[0117] Step S101-14: Determine the target image data corresponding to the acquisition device according to the multiple image frames after the removal, specifically, determine the target image data corresponding to the acquisition device according to the completed image frames, that is, based on the formula 4 in the above step S101-1323 That is, it is the synthetic frame image after removing the detection object in the i-viewing angle. A specific implementation may be to determine the synthetic frame image completed in step S101-1323 as the target image data corresponding to the acquisition device.

[0118] Regarding step S102: input the target image data and the depth map data of the target image data into the neural network in the form of data pairs, obtain the point cloud data of the target image data in the physical world coordinate system, and the pose data of the acquisition device corresponding to the target image data in the physical world coordinate system.

[0119] The purpose of this step is to obtain point cloud data and the pose data of the acquisition device. Among them, the point cloud data can be a data set, a set consisting of multiple three-dimensional coordinate points, and these three-dimensional coordinate points can represent the geometric shape and other properties of the object surface. Each point usually contains three coordinate values ​​of X, Y, and Z, which are used to determine its position in three-dimensional space. Some point cloud data may also contain additional information such as intensity value, color, and incident direction. In this embodiment, the position is mainly used as an example for description. The pose data is used to represent the posture of the acquisition device, which can be the position and direction in three-dimensional space relative to an object, device or other reference object. In this embodiment, it is mainly for the acquisition device, that is, the pose data of the camera or camera relative to the positioning scene.

[0120] The specific implementation process of step S102 may include:

[0121] Step S102-1: Combining the target image data into the image pair in the form of data pairs;

[0122] Step S102-2: forming a depth map pair according to the acquired depth map of the target image data in the image pair;

[0123] Step S102-3: inputting the target image data in the image pair and the depth map corresponding to the target image data in the depth map pair as an input data pair into the neural network, acquiring local point cloud data of the input data pair, and parameters of the acquisition device corresponding to the input data pair;

[0124] Step S102-4: Determine the position data of the acquisition device in the world coordinate system and the global point cloud data of the target image data in the world coordinate system according to the parameters and the local point cloud data.

[0125] like Figure 3 As shown, Figure 3 is a schematic diagram of a structure of a neural network embodiment in a spatial positioning method provided by the present application. In this embodiment, the neural network can be used as follows Figure 3 The model structure shown. The neural network model in this embodiment can be a trained model, and the data set in the training process can use three-dimensional data sets such as ScanNet and CO3Dv2. For the annotation of the data set, a monocular depth estimation network can be used to automatically annotate the video image, and the camera intrinsic parameters can be obtained during the annotation process. The training data can be composed of image pairs of different perspectives with overlapping fields of view obtained by sampling and corresponding point cloud maps. The training of the neural network model is achieved through data annotation and training data.

[0126] The input of the model structure can be: an image pair with a field of view, which can also be called a synthetic frame image pair or a synthetic snapshot pair, that is, { }, , and the depth map obtained by the monocular depth estimation algorithm { } , . The image is matched to { } target image data is also a completed synthetic frame image, and the depth map is compared to { }, the depth map corresponding to the target image data, constitutes the input data pair { } is input into the neural network model.

[0127] The network backbone structure of the neural network model can adopt a general Vision Transform (VIT) network structure, the output layer passes through a common Encoder structure layer, the Encoder structure layer can include 3 layers, and then the output of the Encoder is input into respective Decoder structure layers, the Decoder structure layer can also include 3 layers. After passing through the first layer of Decoder, the purpose of information sharing is achieved by exchanging some features.

[0128] The output of the model structure can be: ,as well as , and also outputs the internal parameters corresponding to the camera And perspective and The transfer matrix between , the corresponding relationship is shown in formula 5:

[0129] Formula 5

[0130] in, Represents a synthetic frame The corresponding depth map , synthetic frame The corresponding depth map , express Perspective , express Perspective , express The camera intrinsics of the view, express The camera intrinsics of the view, (Another representation of the depth map can be called a point map); The corresponding relationship between θ and depth D can be expressed as shown in Formula 6:

[0131] Formula 6

[0132] In getting any { }The depth map and transfer matrix between After that, set The camera pose According to formula 7, the position and orientation of the camera in the physical world coordinate system can be obtained , and the global point cloud . The posture It can be understood as the camera pose determined by the rotation matrix R and translation matrix t with reference to the physical world coordinate system at the i-viewing angle. Formula 7 is as follows:

[0133]

[0134]

[0135] in, is the rotation matrix at the i-viewing angle, is the translation matrix at the i-viewing angle, It is the point cloud data from the perspective of i.

[0136] The above process can be understood as follows: when there are multiple cameras in the monitoring scene, one of the cameras, a, is used as the reference, and the global point cloud data is obtained based on the point cloud data of camera a and the point cloud data of other cameras. According to the relationship between the data pairs, the unified path of the point cloud data can be determined, and the transfer matrix based on camera a is obtained by rotation, translation and other operations, and then the global point cloud data is obtained. For example: Assuming there are 4 cameras, As a benchmark, two pairs of pairs are: and As a data pair, and As a data pair, and As a data pair, and As data pairs, according to the relationship between the data pairs, the point cloud data of a camera in the data pair will be fused and unified into In the point cloud data (this process can be achieved through operations such as rotation and translation, that is, the above formula), the global point cloud data of the camera required in the monitoring scene in the physical world coordinates can be obtained.

[0137] Regarding step S103: determining the positioning information of the target object in the target image data in the physical world coordinate system based on the ground plane and the posture data in the target image data detected by the point cloud data.

[0138] The purpose of step S103 is to determine the location information of the target object in the physical world coordinates, which depends on the ground plane determined according to the point cloud data and the camera's posture data. It is necessary to detect the ground information in the target image data, that is, the ground plane, according to the point cloud data, and a large amount of point cloud data will increase the storage cost, thereby causing a waste of resources. Therefore, the point cloud data can be compressed and / or filtered before the ground plane detection. Therefore, the specific implementation process of step S103 can be combined with Figure 4 and the following description, such as Figure 4 As shown, Figure 4 It is a flowchart of an embodiment of determining the positioning information of a target object in physical world coordinates in a spatial positioning method provided by the present application.

[0139] To reduce the storage cost of point cloud data and avoid waste of resources, in this embodiment, the point cloud data is first compressed and / or filtered. For example, the point cloud data is first uniformly downsampled, 1 / 3 of the points are retained, and then statistically filtered to remove outlier noise caused by errors in neural network depth estimation. For example, 1 / 3 of the points can be randomly selected, or selection requirements can be set according to different scene positioning requirements, and 1 / 3 of the points are selected according to the selection requirements. The filtering can adopt the existing signal processing basic filtering algorithm, for example, by performing statistical analysis on the neighborhood of each point, based on the distance distribution characteristics from the current point to all neighboring points, outliers that do not meet the requirements are filtered out, and these outliers may include outlier noise caused by errors in neural network depth estimation. When processing point cloud data, downsampling can effectively reduce the amount of data and reduce processing time.

[0140] Based on the point cloud data after compression and / or filtering, the specific implementation process of step S103 may include:

[0141] Step S103-1: Determine the predicted point cloud plane according to the point cloud reference points selected in the point cloud data; in this embodiment, the least square method, which may also be called the random sampling consensus algorithm (RANSAC: Random Sample Consensus), can be used to estimate a plane through the selected point cloud reference points. RANSAC (Random Sample Consensus) randomly selects a certain number of points as candidate inliers, and then estimates the parameters of the plane model based on these points. Then calculate the distance from all points to the estimated plane model, and mark the points with a distance less than a certain threshold as inliers. This process is repeated many times, and finally the plane model with the most inliers is selected as the final fitting result. In this embodiment, the least square method (ie: RANSAC) is mainly used as an example for explanation. Therefore, the specific implementation process of step S103-1 may include: randomly selecting point cloud reference points in the point cloud data; fitting the point cloud reference points to the predicted point cloud plane.

[0142] Step S103-2: Determine the distance from the point cloud reference point to the predicted point cloud plane, and count the distance within the reference distance. The number of point cloud reference points within the range; in this embodiment, the reference distance is the distance from the point to the plane, and the value can be determined by experience, for example, the value is 0.001, and the distance is less than or equal to The points within the range are determined as points within the predicted point cloud plane, and the number of the points is counted.

[0143] Step S103-3: Determine whether the number of the point cloud reference points is greater than a set threshold. If so, determine the predicted point cloud plane as the point cloud reference plane. The threshold may also be an empirical value or may be set as required.

[0144] Step S103-4: Iterate and execute the step of determining the predicted point cloud plane. When the number of point cloud reference points in the point cloud reference plane after the iteration is greater than the number of point cloud reference points in the point cloud reference plane before the iteration, update the point cloud reference plane until the number of iterations meets the number requirement and stops iterating. In this embodiment, the number of iterations can be determined based on empirical values, for example, any value between 80 and 150. The point cloud reference plane can be represented by the following equation:

[0145] ;

[0146] Among them, A, B, and C are the normal vectors of the point cloud reference plane; D is a constant; and x, y, and z are the coordinates of the midpoint of the point cloud.

[0147] It should be noted that the example of predicting a plane by the least square method given in this embodiment does not limit the method of predicting a plane by point cloud data, and the predicted point cloud plane can also be determined by the point cloud data through a region growing algorithm. Among them, the region growing algorithm compares the pixel values ​​or depth values ​​of the points in the point cloud data, classifies similar points into the same region, and thus finds the plane part.

[0148] Step S103-5: Determine the positioning information of the target object in the target image data in the world coordinate system according to the updated point cloud reference plane and the position and posture data. The specific implementation process may include:

[0149] Step S103-51: Rotate and / or translate the point cloud reference plane to the same plane as the positioning scene map, and determine it as the ground plane in the physical world coordinate system; because the positioning scene map can be understood as a top view of a two-dimensional plane, it is necessary to rotate and / or translate the point cloud reference plane to make it in the same plane as the positioning scene map. The specific implementation may include:

[0150] Step S103-511: Determine the rotation axis according to the normal vector of the point cloud reference plane and the normal vector of the positioning scene map; for example: transform the equation of the point cloud reference plane The normal vector Calculate the cross product with the normal vector Z (0,0,1) of the XY plane of the positioning scene map and normalize it to get the rotation axis , as shown in the following formula 8:

[0151] Formula 8

[0152] Step S103-512: Determine the cosine and sine of the angle between the normal vector of the point cloud reference plane and the normal vector of the positioning scene map according to the rotation axis; for example: calculate the normal vector The cosine value (cos_angle) and sine value (sin_angle) of the angle with the normal vector Z (or Z axis) are shown in the following formula 9:

[0153]

[0154] Formula 9

[0155] Step S103-513: Determine a rotation matrix according to the rotation axis, the cosine value and the sine value; for example, as shown in the following formulas 10 and 11, where: and For the rotation matrix, the size can be 3×3:

[0156] Formula 10

[0157] Formula 11

[0158] Step S103-514: Rotate the point cloud data and the point cloud reference plane according to the rotation matrix to determine the rotation point cloud reference plane; in this embodiment, the following formula 12 can be used ( is the global point cloud data) and 13:

[0159] Formula 12

[0160]

[0161] Formula 13

[0162] Step S103-515: The rotated point cloud reference plane is translated to the translated point cloud reference plane located at the same plane position as the positioning scene map, and is determined as the ground plane in the physical world coordinate system. In this embodiment, the point cloud in the rotated point cloud reference plane can be translated to achieve the translation of the rotated point cloud reference plane so that the equation of the rotated point cloud reference plane is The distance from the positioning scene map (XY) plane is 0, and the translation vector t is output.

[0163] In combination with the application scenario, the actual application can be matched. For example, in the data stream detection scenario or the heat map construction scenario, the translation point cloud reference plane can be matched with the positioning scene map, so as to facilitate the positioning assistance needs in the subsequent specific application scenarios. Therefore, the specific implementation of the steps S103-515 may include:

[0164] Step S103-5151: Matching the translation point cloud reference plane with the positioning scene map;

[0165] Step S103-5152: Determine the reference plane of the matched point cloud as the ground plane in the physical world coordinate system.

[0166] The matching process can be implemented manually or by machine to perform the translation, scaling, rotation and other operations required in the matching process, and then the translation point cloud reference plane is matched with the positioning scene map in shape and / or direction. In this embodiment, the positioning scene map can be a CAD drawing or a hand-drawn plan view, or of course, a machine-drawn plan view, etc., a map combined with the actual positioning scene. The shape matching can include matching based on contour edges, and can also be matched based on a certain set reference edge or reference point. The specific matching method is not limited. After matching, the transfer matrix can be obtained. , and the scaling parameters . Given the camera pose , and the resulting transfer matrix , the new camera pose can be calculated by the following formula 14 ; The new plane method is calculated by formula 13 .

[0167] It can be understood that not matching the translated point cloud reference plane with the positioning scene map will not affect the subsequent determination of the positioning information of the target object in the physical world coordinate system. This is only an implementation method provided in combination with different application scenarios and is not a necessary method.

[0168] Step S103-52: performing the same rotation and / or translation on the acquisition device according to the rotation and / or translation of the point cloud reference plane;

[0169] Step S103-53: updating the position and posture data of the acquisition device in the physical world coordinate system according to the rotation and / or translation of the acquisition device; in this embodiment, the known camera position and posture ,and , after rotation and / or translation, the updated camera pose is as shown in the following formula 14 (where and It can be understood as a manually calibrated rotation and translation matrix):

[0170] Formula 14

[0171] Step S103-54: Determine the positioning information of the target object in the target image data in the world coordinate system according to the ground plane in the physical world coordinate system and the updated pose data. The specific implementation process can be based on the obtained camera pose data in the physical world coordinate system. or And the ground plane in the physical world coordinate system or , starting from the optical center of the camera, make a ray passing through a pixel point of the image. The intersection of the ray and the ground plane is the location of the pixel point in the physical world coordinate system, where the pixel point can be a pixel point of the target object.

[0172] Therefore, the specific implementation process of step S103-54 may include:

[0173] Step S103-541: determining, according to the position and posture data, a ray extending from the acquisition device to the target object located in the ground plane;

[0174] Step S103-542: Determine the position information of the intersection point between the ray and the ground plane in the physical world coordinate system as the positioning information of the target object in the physical world coordinate system.

[0175] like Figure 5 As shown, Figure 5 A schematic diagram of a positioning embodiment in a spatial positioning method provided in this application. Assume is the pixel coordinate in the image, and the position A of the pixel in space is calculated using the camera intrinsic parameters, as shown in the following formula 15:

[0176] Formula 15

[0177] Camera optical center As the starting point, draw a ray through point A, then the camera parameter method is expressed as the following formula 16:

[0178] Formula 16

[0179] in, Camera parameters.

[0180] Assume that B is the intersection of the ray and the plane. To obtain the coordinates of B, we need to convert the assumption Bring it into the method , and then the coordinates of point B are obtained as:

[0181] Formula 17

[0182] in:

[0183] Formula 18

[0184] The above is a description of the specific implementation process of a spatial positioning method provided by the present application. Through this method, it is no longer constrained by the installation angle, installation quantity, etc. of acquisition devices such as cameras, and the three-dimensional reconstruction of the positioning space can be achieved under the condition of sparse field of view (i.e., the relevant description of the above step S102). Point cloud data can be obtained through three-dimensional reconstruction, and the ground plane and the camera's posture data in the target image data can be detected through the point cloud data, so that the positioning information of the target object in the physical world coordinate system can be determined, which can not only improve the accuracy of positioning but also reduce the cost of positioning. Therefore, the spatial positioning method provided by the present application can realize the positioning of the target object through the acquisition data of any acquisition device in the positioning scene, without the need for other acquisition devices to cooperate. Of course, in combination with the needs or requirements of the positioning scene, multiple acquisition devices can also be used to realize the positioning of the target object. In addition, in the process of three-dimensional reconstruction, the translation point cloud reference plane can also be matched with the positioning scene map in combination with the positioning demand scene, so as to improve the adaptability of the translation point cloud reference plane and the positioning scene map, so as to facilitate the subsequent data processing in the positioning scene.

[0185] The above is a specific description of an embodiment of a spatial positioning method provided by the present application. Corresponding to the embodiment of a spatial positioning method provided above, the present application also discloses an embodiment of a spatial positioning device, please refer to Figure 6 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0186] like Figure 6 As shown, Figure 6 It is a structural schematic diagram of a spatial positioning device provided in the present application; the device includes: a first determination unit 601, an acquisition unit 602, and a second determination unit 603.

[0187] The first determining unit 601 is used to determine target image data corresponding to the acquisition device by removing the detection object from the image data acquired by the acquisition device in the positioning scene.

[0188] The acquisition unit 602 is used to input the target image data and the depth map data of the target image data into the neural network in the form of data pairs, and obtain the point cloud data of the target image data in the physical world coordinate system, and the pose data of the acquisition device corresponding to the target image data in the physical world coordinate system.

[0189] The second determination unit 603 is used to determine the positioning information of the target object in the target image data in the world coordinate system according to the ground plane and the posture data in the target image data detected by the point cloud data.

[0190] The specific implementation process of the first determining unit 601 may include: an acquiring subunit, a first determining subunit, a removing subunit, and a second determining subunit;

[0191] The acquisition subunit is used to acquire the image and video data collected by the acquisition device in the positioning scene according to the set acquisition time;

[0192] The first determination subunit is used to detect multiple image frames in the image video data to determine whether the detection object exists in the multiple image frames;

[0193] The removal subunit is used to remove the detection object in the detection frame when the determination result of the first determination subunit is yes; the removal subunit may include: a judgment subunit and a completion subunit, the judgment subunit is used to determine whether there is a hole area in the multiple image frames according to the pixel information of the multiple image frames obtained; the completion subunit is used to complete the hole area when the determination result of the judgment subunit is yes. The specific implementation process of the judgment subunit may include: a statistics subunit, a pixel determination unit, a synthesis subunit, and a hole determination subunit; the statistics subunit is used to count the number of occurrences of pixels at the same position of the multiple image frames being outside the detection frame; the pixel determination unit is used to determine the average value of pixels at the same position according to the number of occurrences; the synthesis subunit is used to synthesize the multiple image frames into a synthesized frame image according to the pixel average; the hole determination subunit is used to determine whether there is a detection frame area with empty pixel values ​​in the synthesized frame image, and if so, determine that there is a hole area in the synthesized frame image. The completion subunit includes: an area acquisition subunit and a selection subunit, wherein the area acquisition subunit is used to acquire the area corresponding to the detection frame in the multiple image frames and determine the detection frame area set; the selection subunit is used to select the target detection frame that meets the selection requirement from the detection frame area set according to the set area selection requirement; the completion subunit is specifically used to complete the hole area in the synthetic frame image according to the pixels at the corresponding position in the image frame where the target detection frame is located;

[0194] The second determination subunit is used to determine the target image data corresponding to the acquisition device according to the multiple image frames after the removal. Specifically, the target image data corresponding to the acquisition device can be determined according to the completed image frame, that is, the completed synthetic frame image is determined as the target image data corresponding to the acquisition device.

[0195] The specific implementation process of the acquisition unit 602 may include: a first composition subunit, a second composition subunit, an acquisition subunit and a determination subunit;

[0196] The first composition subunit is used to compose the target image data into the image pair in the form of data pairs;

[0197] The second composition subunit is used to compose a depth map pair according to the acquired depth map of the target image data in the image pair;

[0198] The acquisition subunit is used to input the target image data in the image pair and the depth map corresponding to the target image data in the depth map pair as an input data pair into the neural network, and acquire the local point cloud data of the input data pair and the parameters of the acquisition device corresponding to the input data pair;

[0199] The determination subunit is used to determine the posture data of the acquisition device in the world coordinate system and the global point cloud data of the target image data in the world coordinate system according to the parameters and the local point cloud data.

[0200] The specific implementation process of the determination unit 603 may include: a first determination subunit, a statistics subunit, a second determination subunit, an iteration subunit and a third determination subunit;

[0201] The first determination subunit is used to determine the predicted point cloud plane based on the point cloud reference points selected in the point cloud data; the specific implementation process of the first determination subunit may include: a selection subunit and a fitting subunit; the group selection subunit is used to randomly select point cloud reference points in the point cloud data; the fitting subunit is used to fit the point cloud reference points selected in the selection subunit into the predicted point cloud plane.

[0202] The statistical subunit is used to determine the distance from the point cloud reference point to the predicted point cloud plane, and to count the distance between the point cloud reference point and the predicted point cloud plane. The number of reference points of the point cloud within the range;

[0203] The second determining subunit is used to determine whether the number of the point cloud reference points is greater than a set threshold, and if so, determine the predicted point cloud plane as the point cloud reference plane;

[0204] The iterative subunit is used to iteratively execute the step of determining the predicted point cloud plane, and when the number of point cloud reference points in the point cloud reference plane after iteration is greater than the number of point cloud reference points in the point cloud reference plane before iteration, update the point cloud reference plane until the number of iterations meets the number requirement and stops iterating;

[0205] The third determination subunit is used to determine the positioning information of the target object in the target image data in the world coordinate system according to the updated point cloud reference plane and the position and posture data. The specific implementation process of the third determination subunit may include: a first rotation / translation subunit, a second rotation / translation subunit, an updating subunit and a determination subunit;

[0206] The first rotation / translation subunit is used to determine the point cloud reference plane rotated and / or translated to the same plane as the positioning scene map as the ground plane in the physical world coordinate system; the specific implementation process may include: a rotation axis determination subunit, used to determine the rotation axis according to the normal vector of the point cloud reference plane and the normal vector of the positioning scene map; an angle value determination subunit, used to determine the cosine value and sine value of the angle between the normal vector of the point cloud reference plane and the normal vector of the positioning scene map according to the rotation axis determined in the rotation axis determination subunit; a matrix determination subunit, used to determine the rotation axis determined by the rotation axis determination subunit and the cosine value and sine value determined in the angle value determination subunit. A rotation matrix; a rotation subunit, used to rotate the point cloud data and the point cloud reference plane according to the rotation matrix determined in the matrix determination subunit, and determine the rotated point cloud reference plane; a translation determination subunit, used to determine the translated point cloud reference plane after translating the rotated point cloud reference plane determined by the rotation subunit to the same plane position as the positioning scene map as the ground plane in the physical world coordinate system; the specific implementation process of the translation determination subunit may include: a matching subunit and a determination subunit; the matching subunit is used to match the translated point cloud reference plane with the positioning scene map; the determination subunit determines the matched point cloud reference plane after matching as the ground plane in the physical world coordinate system;

[0207] The second rotation / translation subunit is used to perform the same rotation and / or translation on the acquisition device according to the rotation and / or translation of the point cloud reference plane;

[0208] The updating subunit is used to update the position and posture data of the acquisition device in the physical world coordinate system according to the rotation and / or translation of the acquisition device;

[0209] The determination subunit is used to determine the positioning information of the target object in the target image data in the world coordinate system according to the ground plane in the physical world coordinate system and the updated posture data.

[0210] The specific implementation process of the determination unit 603 may include: a ray determination subunit and a positioning determination subunit; the ray determination subunit is used to determine the ray extending from the acquisition device to the target object located in the ground plane according to the posture data; the ray determination subunit is used to determine the position information of the intersection point between the ray and the ground plane in the physical world coordinate system as the positioning information of the target object in the physical world coordinate system.

[0211] The above is a description of the specific implementation process of a spatial positioning device provided in this application. This process corresponds to the implementation process of the above-mentioned spatial positioning method. Therefore, the details of the specific implementation process of the device can refer to the content of the method and will not be described in detail here.

[0212] Based on the above, this application also provides a method for constructing a heat map, such as Figure 7 As shown, Figure 7 This is a flow chart of a method for constructing a heat map provided by the present application, the method comprising:

[0213] Step S701: According to the above-mentioned spatial positioning method, the positioning information of the target object in the target image data collected by the acquisition device in the physical world coordinate system is obtained;

[0214] Step S702: Detecting a moving object within a preset range of the target object according to the positioning information;

[0215] Step S703: construct a heat map of the target object according to the number of moving objects.

[0216] In this embodiment, the heat map construction method can be applied to the electronic platform of a 4S store, or the electronic platform of thermal data in an exhibition hall, or the display platform for crowd guidance and observation in tourist attractions, the analysis platform for passenger flow in shopping malls, etc.; thereby providing an important basis for subsequent management planning, data analysis, etc. of exhibits, items, passenger flow, etc.

[0217] Accordingly, the present application also provides a thermal map construction device, such as Figure 8 As shown, Figure 8 It is a structural schematic diagram of a thermal map construction device provided by the present application, and the device includes: an acquisition unit 801, a detection unit 802 and a construction unit 803; the acquisition unit 801 is used to obtain the positioning information of the target object in the target image data collected by the acquisition device in the physical world coordinate system according to the above-mentioned spatial positioning method; the detection unit 802 is used to detect the moving objects within the preset range of the target object according to the positioning information; the construction unit 803 is used to construct the thermal map of the target object according to the number of moving objects.

[0218] The specific implementation process of the thermal map construction device can also refer to the specific contents of the above-mentioned spatial positioning method and thermal map construction method, which will not be described in detail here.

[0219] Based on the above, the present application also provides a computer storage medium, including: a computer program, when the computer program is run on an electronic device, the electronic device executes the spatial positioning method as described above, or executes the thermal map construction method as described above.

[0220] Based on the above, the present application also provides an electronic device, such as Fig. 9 As shown, Fig. 9 9 is a schematic diagram of the structure of an electronic device provided by the present application, the electronic device comprising: a processor 901 and a memory 902;

[0221] The memory 902 is used to store a program for processing data generated by the electronic device. When the program is read and executed by the processor 901, the program executes the above-mentioned spatial positioning method, or the above-mentioned thermal map construction method.

[0222] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0223] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0224] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0225] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0226] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0227] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A spatial positioning method, characterized in that: include: Determine target image data corresponding to the acquisition device by removing the detection object from the image data acquired by the acquisition device in the positioning scene; Inputting the target image data and the depth map data of the target image data into the neural network in the form of a data pair, obtaining the point cloud data of the target image data in the physical world coordinate system, and the position and posture data of the acquisition device corresponding to the target image data in the physical world coordinate system; including: composing the target image data into an image pair in the form of a data pair, composing a depth map pair according to the depth map of the target image data in the image pair, inputting the target image data in the image pair and the depth map corresponding to the target image data in the depth map pair as input data pairs into the neural network, obtaining the local point cloud data of the input data pair, and the parameters of the acquisition device corresponding to the input data pair, and determining the position and posture data of the acquisition device in the world coordinate system and the global point cloud data of the target image data in the world coordinate system according to the parameters and the local point cloud data; According to the ground plane and the posture data in the target image data detected by the point cloud data, the positioning information of the target object in the target image data in the physical world coordinate system is determined.

2. The spatial positioning method according to claim 1, characterized in that: The method of removing the detection object from the image data acquired by the acquisition device in the positioning scene to determine the target image data corresponding to the acquisition device includes: Acquiring the image and video data collected by the acquisition device in the positioning scene according to the set acquisition time; Detecting a plurality of image frames in the image video data to determine whether the detection object exists in the plurality of image frames; If so, remove the detection object in the detection frame; According to the multiple image frames after the removal, target image data corresponding to the acquisition device is determined.

3. The spatial positioning method according to claim 2, characterized in that: The removing of the detection object comprises: Determine whether a hole area exists in the multiple image frames according to the acquired pixel information of the multiple image frames; If yes, then fill the hole area; The step of determining target image data corresponding to the acquisition device according to the removed multiple image frames includes: According to the completed image frame, the target image data corresponding to the acquisition device is determined.

4. The spatial positioning method according to claim 3, characterized in that: The step of determining whether a hole area exists in the plurality of image frames according to the acquired pixel information of the plurality of image frames comprises: Counting the number of times that pixels at the same position of the multiple image frames are located outside the detection frame; Determine the mean value of pixels at the same position according to the number of occurrences; Combining the multiple image frames into a composite frame image according to the pixel mean value; Determine whether there is a detection frame area with empty pixel values ​​in the synthesized frame image, and if so, determine that there is a hole area in the synthesized frame image.

5. The spatial positioning method according to claim 4, characterized in that: The filling of the hole area includes: Acquire areas corresponding to the detection frames in the multiple image frames to determine a set of detection frame areas; According to the set area selection requirement, a target detection frame that meets the selection requirement is selected from the detection frame area set; Completing the hole area in the synthesized frame image according to pixels at corresponding positions in the image frame where the target detection frame is located; The step of determining the target image data corresponding to the acquisition device according to the completed image frame includes: The completed synthetic frame image is determined as the target image data corresponding to the acquisition device.

6. The spatial positioning method according to claim 1, characterized in that: The determining, based on the ground plane and the posture data in the target image data detected by the point cloud data, the positioning information of the target object in the target image data in the physical world coordinate system comprises: Determining a predicted point cloud plane according to a point cloud reference point selected from the point cloud data; Determine the distance from the point cloud reference point to the predicted point cloud plane, and count the distances between the point cloud reference point and the predicted point cloud plane. The number of reference points of the point cloud within the range; Determine whether the number of the point cloud reference points is greater than a set threshold, and if so, determine the predicted point cloud plane as the point cloud reference plane; Iteratively executing the step of determining the predicted point cloud plane, when the number of point cloud reference points in the point cloud reference plane after the iteration is greater than the number of point cloud reference points in the point cloud reference plane before the iteration, updating the point cloud reference plane until the number of iterations meets the number requirement and stops the iteration; According to the updated point cloud reference plane and the position and posture data, the positioning information of the target object in the target image data in the world coordinate system is determined.

7. The spatial positioning method according to claim 6, characterized in that: The step of determining the positioning information of the target object in the target image data in the world coordinate system according to the updated point cloud reference plane and the pose data includes: A point cloud reference plane that is rotated and / or translated to be in the same plane as the positioning scene map is determined as the ground plane in the physical world coordinate system; According to the rotation and / or translation of the point cloud reference plane, performing the same rotation and / or translation on the acquisition device; According to the rotation and / or translation of the acquisition device, updating the position and posture data of the acquisition device in the physical world coordinate system; According to the ground plane in the physical world coordinate system and the updated posture data, the positioning information of the target object in the target image data in the world coordinate system is determined.

8. The spatial positioning method according to claim 7, characterized in that: The step of rotating and / or translating the point cloud reference plane that is in the same plane as the positioning scene map to determine the ground plane in the physical world coordinate system includes: Determining a rotation axis according to a normal vector of the point cloud reference plane and a normal vector of the positioning scene map; Determine the cosine value and sine value of the angle between the normal vector of the point cloud reference plane and the normal vector of the positioning scene map according to the rotation axis; Determine a rotation matrix according to the rotation axis, the cosine value and the sine value; Rotate the point cloud data and the point cloud reference plane according to the rotation matrix to determine a rotated point cloud reference plane; The rotated point cloud reference plane is translated to a translated point cloud reference plane that is located at the same plane position as the positioning scene map, and is determined as the ground plane in the physical world coordinate system.

9. The spatial positioning method according to claim 8, characterized in that: The step of translating the rotation point cloud reference plane to a translated point cloud reference plane located at the same plane position as the positioning scene map, and determining the translated point cloud reference plane as the ground plane in the physical world coordinate system, comprises: Matching the translated point cloud reference plane with the positioning scene map; The matched point cloud reference plane is determined as the ground plane in the physical world coordinate system.

10. The spatial positioning method according to claim 6, characterized in that: The step of determining a predicted point cloud plane according to selecting a reference point of the point cloud in the point cloud data comprises: Randomly selecting a point cloud reference point in the point cloud data; The point cloud reference points are fitted into a predicted point cloud plane.

11. The spatial positioning method according to claim 1, characterized in that: The determining, based on the ground plane and the posture data in the target image data detected by the point cloud data, the positioning information of the target object in the target image data in the world coordinate system comprises: Determining, according to the position and posture data, a ray extending from the acquisition device to the target object located in the ground plane; The position information of the intersection point between the ray and the ground plane in the physical world coordinate system is determined as the positioning information of the target object in the physical world coordinate system.

12. A spatial positioning device, characterized in that: include: A first determining unit, configured to determine target image data corresponding to the acquisition device by removing the detection object from the image data acquired by the acquisition device in the positioning scene; An acquisition unit is used to input the target image data and the depth map data of the target image data into the neural network in the form of a data pair, and obtain the point cloud data of the target image data in the physical world coordinate system, and the posture data of the acquisition device corresponding to the target image data in the physical world coordinate system; including: forming an image pair with the target image data in the form of a data pair, forming a depth map pair according to the depth map of the target image data in the image pair, inputting the target image data in the image pair and the depth map corresponding to the target image data in the depth map pair as input data pairs into the neural network, obtaining local point cloud data of the input data pair, and parameters of the acquisition device corresponding to the input data pair, and determining the posture data of the acquisition device in the world coordinate system and the global point cloud data of the target image data in the world coordinate system according to the parameters and the local point cloud data; The second determining unit is used to determine the positioning information of the target object in the target image data in the world coordinate system according to the ground plane and the posture data in the target image data detected by the point cloud data.

13. A method for constructing a heat map, characterized in that: include: According to the spatial positioning method described in any one of claims 1 to 11 above, the positioning information of the target object in the physical world coordinate system in the target image data acquired by the acquisition device is obtained; Detecting a moving object within a preset range of the target object according to the positioning information; According to the number of moving objects, a heat map of the target object is constructed.

14. A computer storage medium, characterized in that: It comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the spatial positioning method as described in any one of claims 1 to 11, or the thermal map construction method as described in claim 13.

15. An electronic device, characterized in that: include: processor; A memory for storing a program for processing data generated by an electronic device, wherein when the program is read and executed by the processor, the program executes the spatial positioning method as described in any one of claims 1 to 11, or the thermal map construction method as described in claim 13.

Images