Target positioning method and device, computer device and storage medium

Abstract

The application relates to a target positioning method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a to-be-processed image; calculating a current pose of a self-moving device in a target map coordinate system according to the to-be-processed image; identifying a target in the to-be-processed image and acquiring a position of the identified target in a camera coordinate system; and calculating a position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system. The position of the target in the target map can be obtained by using the method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a target positioning method and device, computer equipment, a storage medium and a computer program product. BACKGROUND

[0002] Since substations are mostly located in sparsely populated or even harsh environments, and substations usually occupy a large area, if only artificial inspection is used, the efficiency is low and the reliability is poor, therefore, inspection robots are used to efficiently and continuously inspect high-voltage and unattended substation equipment. At present, the navigation map of the inspection robot is a 2D map, and the robot cannot realize functions such as climbing slopes and steps, and the inspection robot can only detect targets during the inspection process, but cannot know the positions of the targets in the map. SUMMARY

[0003] Therefore, it is necessary to provide a target positioning method, device, computer equipment, computer readable storage medium and computer program product capable of obtaining the position of a target in a target map to solve the above technical problems.

[0004] In a first aspect, the present application provides a target positioning method. The method comprises:

[0005] obtaining a to-be-processed image;

[0006] calculating a current pose of a self-moving device in a target map coordinate system according to the to-be-processed image;

[0007] identifying a target in the to-be-processed image, and obtaining a position of the identified target in a camera coordinate system;

[0008] calculating a position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system.

[0009] In one embodiment, the identification of the target in the to-be-processed image and the obtaining of the position of the identified target in the camera coordinate system comprise:

[0010] determining whether the to-be-processed image includes the target;

[0011] when the to-be-processed image includes the target, identifying a type of the target and a bounding box of the target in the to-be-processed image;

[0012] calculating the position of the target in the camera coordinate system according to the bounding box of the target in the to-be-processed image and a depth image.

[0013] In one embodiment, before the obtaining of the to-be-processed image, the method further comprises:

[0014] reconstructing a target map according to the key frame data.

[0015] In one embodiment, the method further comprises, before reconstructing a target map according to the key frame data:

[0016] acquiring a current frame image by using a depth camera;

[0017] matching the current frame image with a previous frame image;

[0018] when a matching result of features of the current frame image and the previous frame image meets a preset condition, taking the current frame image as a current key frame image;

[0019] storing information of the current key frame image as key frame data.

[0020] In one embodiment, when the matching result of features of the current frame image and the previous frame image meets the preset condition, taking the current frame image as a current key frame image, the method further comprises:

[0021] calculating a target pose of the self-moving device corresponding to the current key frame image;

[0022] adjusting a current pose of the self-moving device according to the target pose of the self-moving device corresponding to the current key frame image, and acquiring a next frame image by using the depth camera as a current frame image, and continuing the matching of the current frame image with the previous frame image.

[0023] In one embodiment, the method further comprises:

[0024] when the matching result of features of the current frame image and the previous frame image does not meet the preset condition, searching for a key frame image matching the current frame image from the stored key frame images.

[0025] In a second aspect, the application further provides a target positioning device. The device comprises:

[0026] an image acquisition module, configured to acquire a to-be-processed image;

[0027] a pose calculation module, configured to calculate a current pose of a self-moving device in a target map coordinate system according to the to-be-processed image;

[0028] a position calculation module, configured to identify a target in the to-be-processed image, and acquire a position of the identified target in a camera coordinate system;

[0029] The target position calculation module is configured to calculate the position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system.

[0030] In a third aspect, the present application provides a computer device. The computer device comprises a memory and a processor. The memory stores a computer program. The processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0031] In a fourth aspect, the present application provides a computer readable storage medium. The computer readable storage medium stores a computer program. The computer program, when executed by a processor, implements the steps of the method in any one of the above embodiments.

[0032] In a fifth aspect, the present application provides a computer program product. The computer program product comprises a computer program. The computer program, when executed by a processor, implements the steps of the method in any one of the above embodiments.

[0033] The target positioning method, device, computer device, storage medium and computer program product described above can obtain the current pose of the self-moving device in the target map coordinate system and the position of the recognized target in the camera coordinate system by processing the to-be-processed image, and can obtain the position of the target in the target map and complete the navigation positioning function without laser emitters and receivers by combining the current pose of the self-moving device in the target map coordinate system and the position of the recognized target in the camera coordinate system. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 FIG. 1 is a diagram of an application environment of the target positioning method in one embodiment;

[0035] Figure 2 FIG. 2 is a flowchart of the target positioning method in one embodiment;

[0036] Figure 3 FIG. 3 is a diagram of a bounding box of a target in a to-be-processed image in one embodiment;

[0037] Figure 4 FIG. 4 is a flowchart of the target positioning method in another embodiment;

[0038] Figure 5 FIG. 5 is a structural block diagram of the target positioning device in one embodiment;

[0039] Figure 6 FIG. 6 is an internal structure diagram of the computer device in one embodiment. DETAILED DESCRIPTION

[0040] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0041] The target positioning method provided by the embodiments of the present application can be applied to an application environment as shown in Figure 1 . The self-moving device 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on a cloud or other network server. The server 104 obtains a to-be-processed image photographed by the self-moving device 102; calculates a current pose of the self-moving device in a target map coordinate system according to the to-be-processed image; identifies a target in the to-be-processed image, and obtains a position of the identified target in a camera coordinate system; and calculates a position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system, to obtain the coordinates of the specific target in the map and complete the navigation positioning function without laser emitters and receivers. The self-moving device 102 can be, but is not limited to, various robots, unmanned aerial vehicles and other remotely controllable devices, which can replace technical personnel to collect information in sparsely populated areas or even harsh environments. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0042] In one embodiment, as shown in Figure 2 , a target positioning method is provided. The method is described below by taking a server in Figure 1 as an example, which includes the following steps:

[0043] S202, obtaining a to-be-processed image.

[0044] Specifically, the to-be-processed image is an image photographed by a self-moving device, which can include a color image and / or a depth image. The color image refers to an image obtained by changing and superimposing three color channels of red, green and blue to obtain various colors. The depth image refers to an image that can reflect information related to the distance of the surface of a scene object.

[0045] Optionally, the self-moving device can be a robot carrying a depth camera, which refers to a camera that can measure the distance (depth) from an object to the camera. The robot uses the depth camera to photograph images of the surrounding environment during movement. Preferably, the to-be-processed image photographed by the depth camera includes a color image and a depth image.

[0046] S204, calculating a current pose of the self-moving device in a target map coordinate system according to the to-be-processed image.

[0047] The target map refers to a map of a specific region, for example, a 3D map. The target map can be generated by the server in advance by capturing images of a specific region by a mobile device. The target map can be a 3D map of any region reconstructed according to an actual scene. The target map coordinate system refers to a three-dimensional rectangular coordinate system established with the position of the mobile device as the origin when the mobile device captures an image that meets a preset condition. For example, when a camera carried by the mobile device captures a frame of image with a feature point reaching a preset number of points, the current position of the mobile device is taken as the origin to establish a three-dimensional rectangular coordinate system. The current pose refers to the position and angle of the camera capturing the current image to be processed.

[0048] Specifically, the server calculates the current position of the mobile device in the target map coordinate system according to the image to be processed captured by the depth camera. For example, the server can obtain the pose of the robot in the substation map, i.e., the position of the robot in the substation map and the angle of capturing the image to be processed, by processing the image to be processed. The substation map refers to a target map in a specific use scenario. Optionally, the RGBD data captured by the depth camera can be processed using a SLAM algorithm (Simultaneous Localization and Mapping) to obtain the current pose of the robot.

[0049] In S206, a target in the image to be processed is recognized, and the position of the recognized target in the camera coordinate system is obtained.

[0050] Specifically, the target refers to each object included in the image to be processed, such as a vehicle, a pedestrian, and the like. When the target is recognized, the target can also be classified to obtain targets of different categories. The camera coordinate system refers to a three-dimensional rectangular coordinate system established with the depth camera as the origin.

[0051] Specifically, the server first recognizes the target in the image to be processed, and then processes the image to be processed to obtain the position of the recognized target in the camera coordinate system. For example, the position of the target in the camera coordinate system is obtained by combining the position of the target in the color image and the pixel value in the corresponding depth image. Optionally, the server can first determine whether the color image in the image to be processed includes a target. If the color image includes a target, the label and the bounding box of the target are recognized, and the position of the target in the color image is obtained according to the bounding box. The position of the target in the camera coordinate system is further obtained by combining the depth image included in the image to be processed. Optionally, the server can use a pre-trained target detection algorithm to recognize the target in the image to be processed.

[0052] S208, according to the current pose of the mobile device and the position of the target in the camera coordinate system, the position of the target in the target map coordinate system is calculated.

[0053] Specifically, the server processes the current pose of the mobile device and the position of the target in the camera coordinate system to obtain the position of the target in the target map coordinate system, wherein the server can obtain the position of the target in the target map coordinate system by coordinate transformation.

[0054] Specifically, since the current pose is in the target map coordinate system, the conversion relationship between the origin and the current pose corresponding to the mobile device in the camera coordinate system can be calculated, and the target in the camera coordinate system can be converted to the target map coordinate system according to the conversion relationship.

[0055] In the above embodiment, the current pose of the mobile device in the target map coordinate system and the position of the identified target in the camera coordinate system are obtained by processing the to-be-processed image, and the position of the target in the target map coordinate system can be obtained by combining the current pose of the mobile device in the target map coordinate system and the position of the identified target in the camera coordinate system, without laser emitter and receiver, and the navigation positioning function can be completed.

[0056] In one embodiment, the target in the to-be-processed image is identified, and the position of the identified target in the camera coordinate system is obtained, including: judging whether the to-be-processed image includes the target; when the to-be-processed image includes the target, identifying the type of the target and the bounding box of the target in the to-be-processed image; and calculating the position of the target in the camera coordinate system according to the bounding box of the target in the to-be-processed image and the depth image.

[0057] The bounding box refers to a frame with a regular shape generated according to the outer contour line of the target in the to-be-processed image, and the specific combination of the bounding box and the depth image is shown in FIG. 4. Figure 3 Figure 3 The black solid line in FIG. 4 is the bounding box of the target (vehicle) in the to-be-processed image.

[0058] ​Specifically, after receiving the to-be-processed image, the server first determines whether the to-be-processed image includes a target, when the server identifies that the to-be-processed image includes a target, identifies the type of the target and the bounding box of the target in the to-be-processed image, and then further calculates the position of the target in the camera coordinate system according to the bounding box and the depth image of the target in the to-be-processed image. Optionally, the server can use a pre-trained target detection algorithm to process the color image included in the to-be-processed image to identify the type of the target and the bounding box of the target in the to-be-processed image. In an embodiment, the server can input the training image, the target label of the target to be identified in the image, and the target bounding box into a deep learning model, wherein the deep learning model refers to a machine training model capable of obtaining a target label and a target bounding box in an image. Optionally, the deep learning model can be YOLOv3 (You Only Look Once, an object detection algorithm). The deep learning model is trained and learned by a large number of training images, target labels, and target bounding boxes to obtain a target detection model capable of processing the to-be-processed image to identify the target and the target bounding box in the image. Preferably, the server can use a pre-selected target detection algorithm to determine whether the color image includes a target. If the to-be-processed image includes a target, the target label and the bounding box of the target in the image are given. According to the bounding box, the coordinates (x p ,y p ) of the target in the color image can be calculated, and then combined with the depth image in the to-be-processed image to calculate, for example, coordinate conversion, to obtain the position (X P ,Y P ,Z P ) of the target in the camera coordinate system.

[0059] In the above embodiment, the bounding box of the target in the to-be-processed image can be used to obtain the coordinates of the target in the to-be-processed image, and the depth image in the to-be-processed image can be used to calculate the position of the target in the camera coordinate system. The position of the target in the camera coordinate system can be used for subsequent calculation of the position of the target in the target map.

[0060] In one embodiment, before obtaining the to-be-processed image, the method further comprises: reconstructing a target map according to key frame data.

[0061] The key frame refers to an image frame including important features required when reconstructing a map, and the key frame data refers to data including key frame information, which can be a color image and a depth image corresponding to the key frame.

[0062] Specifically, scene reconstruction is needed to obtain a target map according to the key frame data before processing the image to be processed, and optionally, an octree map can be generated as the target map after generating a point cloud map using the key frame data. In an embodiment, first, the inspection robot is placed at any position in the substation, the inspection robot carries a depth camera, the inspection robot is remotely controlled to move in the substation, the depth camera collects color images and depth images of the surrounding environment during the movement, then the color images and the depth images are processed to obtain key frame data, and then the substation map is reconstructed according to the key frame data, and optionally, the color images can be preprocessed to extract color features and match the previous frame image to determine whether it is a key frame.

[0063] In the above embodiment, scene reconstruction can be performed according to the key frame data to obtain a target map, and the position of the target can be determined through the target map, and the self-moving device can complete the task of climbing the stairs.

[0064] In one of the embodiments, before scene reconstruction is performed according to the key data to obtain a target map, it includes: collecting a current frame image by a depth camera; matching the current frame image with a previous frame image; when the matching result of the features of the current frame image and the previous frame image meets a preset condition, taking the current frame image as a current key frame image; and saving the current key frame image information as key frame data.

[0065] Specifically, the server needs to obtain key frame data before scene reconstruction is performed according to the key frame data to obtain a target map, and then reconstructs the scene according to the key frame data. Specifically, the server first obtains a current frame image collected by a depth camera, wherein the current frame image includes a depth image and a color image, then matches the current frame image with a previous frame image, and optionally, matches the color image included in the current frame with the color image included in the previous frame image to determine whether the current frame is a key frame, and preferably, pre-processes the color image included in the current frame image and matches it with the previous frame image, which can speed up the matching process. In other embodiments, the preprocessing can be any one of graying and feature extraction. Specifically, when the feature matching of the current frame image and the previous frame image meets a preset condition, the current frame image is taken as a current key frame image, and optionally, the preset condition can be any one of 1) matching with the previous frame is successful, that is, the number of matching points of the adjacent two frames is greater than 10; 2) the interval with the previous key frame is greater than 20 frames; 3) new feature points appear, wherein the preset condition can be adjusted according to the actual situation, and then the current key frame information is saved as key frame data, and optionally, the key frame information is saved as a binary file, and the corresponding depth image and color image of each key frame are also saved.

[0066] In one embodiment, the server converts the color image into a grayscale image, extracts ORB features, and provides the grayscale image and the extracted features to the SLAM algorithm system. After the robot is started, the SLAM algorithm system enters an initialization stage. When the number of feature points in the current frame is greater than 500, the camera position of the current frame is set as the coordinate origin. After successful initialization, the SLAM algorithm system calculates the ORB features in each frame and performs ORB feature matching with the previous frame. When the matching meets the preset condition, the current frame is set as a key frame. The key frame information is saved as a binary file, and the color image and the depth image corresponding to each key frame are saved. The key frame information includes the pose of the current key frame.

[0067] In the above embodiment, whether the current frame is a key frame can be determined by matching with the previous frame. This can make the server reconstruct the target map faster when using key frame data.

[0068] In one embodiment, when the matching result of the features of the current frame and the previous frame meets the preset condition, the current frame is set as a current key frame. The method further includes calculating the target pose of the self-moving device corresponding to the current key frame, adjusting the current pose of the self-moving device according to the target pose of the self-moving device corresponding to the current key frame, and collecting the next frame as the current frame by the depth camera, and continuing to match the current frame with the previous frame.

[0069] The target pose refers to the pose required by the self-moving device when capturing the next frame. After the server determines that the current frame is a key frame, the pose of the current key frame is calculated. The pose of the current key frame can be estimated by the PnP algorithm (Perspective-n-Point, a method for solving 3D to 2D point motion). The target pose can be calculated by the BundleAdjustment algorithm. The self-moving device adjusts the current pose according to the target pose, continues to capture the next frame by the depth camera carried by the self-moving device, and takes the next frame as the current frame for processing by the server. The current frame is matched with the previous frame, and the process is repeated until a series of key frame images and key frame information are obtained.

[0070] In the above embodiment, the pose of the self-moving device is continuously adjusted to obtain a series of key frames for reconstructing the target map.

[0071] In one of the embodiments, the target positioning method further comprises: when the matching result of the features of the current frame image and the previous frame image does not satisfy the preset condition, searching the stored key frame image for a key frame matching the current frame image.

[0072] Specifically, when the matching result of the features of the current frame image and the previous frame image does not satisfy the preset condition, i.e., there are too few features between the two frames of images, the server considers that a tracking failure occurs at this time, and thus the server enters a repositioning to find a key frame matching the current frame image for repositioning. Specifically, a key frame matching the current frame image can be searched from the stored key frame images, and the repositioning is successful when it is confirmed that the current frame image is a certain key frame in the stored key frame images. Optionally, the server can accelerate the matching process of the current frame image and the stored key frame image by continuously calculating the BOW value of the current frame.

[0073] In the above embodiments, repositioning can be performed by image matching to avoid the case that the key frame data collection is incomplete.

[0074] In one of the embodiments, the target positioning method further comprises: Figure 4 , Figure 4 Fig. 1 is a flowchart of a process of calculating the position of a target in a target map by a server through RGBD data captured by a depth camera in one of the embodiments, comprising the following steps:

[0075] The server first provides the RGBD data captured by the depth camera carried by the robot in real time to the SLAM algorithm and the YOLOv3 algorithm at the same time. After the RGBD data is processed by the SLAM algorithm, the current pose (X R ,Y R ,Z R ,Q R ) of the robot is obtained. The YOLOv3 first judges the RGB image, and when it identifies that the RGB image includes a target, the label of the target and the boudingbox of the target in the image are given. According to the boudingbox, the coordinates (x p ,y p ) of the target O in the RGB image can be calculated, and then the coordinates (X p ,Y P ,Z P ) of the target O in the camera coordinate system can be obtained in combination with the depth image. At this time, the position of the target O in the map can be calculated in combination with the current pose of the robot.

[0076] In the above embodiments, the RGBD data captured by the depth camera is processed by the SLAM algorithm and the YOLOv3 algorithm to obtain the position of the target in the target map.

[0077] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0078] In one embodiment, as shown in FIG. 1, a target positioning apparatus is provided, comprising an image acquisition module 100, a pose calculation module 200, a position calculation module 300, and a target position calculation module 400, wherein: Figure 5

[0079] The image acquisition module 100 is configured to acquire a to-be-processed image.

[0080] The pose calculation module 200 is configured to calculate a current pose of a mobile device in a target map coordinate system according to the to-be-processed image.

[0081] The position calculation module 300 is configured to identify a target in the to-be-processed image and acquire a position of the identified target in a camera coordinate system.

[0082] The target position calculation module 400 is configured to calculate a position of the target in the target map coordinate system according to the current pose of the mobile device and the position of the target in the camera coordinate system.

[0083] In one embodiment, the position calculation module 300 described above comprises:

[0084] A judgment unit is configured to judge whether the to-be-processed image includes a target.

[0085] A bounding box acquisition unit is configured to identify a type of the target and a bounding box of the target in the to-be-processed image when the to-be-processed image includes the target.

[0086] A camera coordinate system calculation unit is configured to calculate the position of the target in the camera coordinate system according to the bounding box of the target in the to-be-processed image and a depth image.

[0087] In one embodiment, the target positioning apparatus further comprises:

[0088] A map reconstruction module is configured to reconstruct a target map according to key frame data.

[0089] ​In one embodiment, the map reconstruction module comprises:

[0090] An image acquisition unit is configured to acquire a current frame image by using the depth camera.

[0091] An image matching unit is configured to match the current frame image with a previous frame image.

[0092] A key frame calculation unit is configured to take the current frame image as a current key frame image when a matching result of features of the current frame image and the previous frame image meets a preset condition.

[0093] A key frame information saving unit is configured to save information of the current key frame image as key frame data.

[0094] In one embodiment, the map reconstruction module further comprises:

[0095] A target pose calculation unit is configured to calculate a target pose of the self-moving device corresponding to the current key frame image.

[0096] A key frame matching unit is configured to adjust a current pose of the self-moving device according to the target pose of the self-moving device corresponding to the current key frame image, and continue to match the current frame image with the previous frame image by acquiring a next frame image as the current frame image by using the depth camera.

[0097] In one embodiment, the map reconstruction module further comprises:

[0098] A key frame searching unit is configured to search for a key frame matching the current frame image from the stored key frame images when the matching result of features of the current frame image and the previous frame image does not meet the preset condition.

[0099] Each module in the target positioning device can be realized by software, hardware and a combination thereof in whole or in part. Each module can be embedded in or independent of a processor in a computer device in a hardware form, or can be stored in a memory in a computer device in a software form, so as to call and execute the operations corresponding to each module by a processor.

[0100] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 6As shown in the figure. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data to be processed. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a target positioning method.

[0101] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0102] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the following steps: obtaining an image to be processed; calculating a current pose of a mobile device in a target map coordinate system according to the image to be processed; identifying a target in the image to be processed and obtaining the position of the identified target in a camera coordinate system; and calculating the position of the target in the target map coordinate system according to the current pose of the mobile device and the position of the target in the camera coordinate system.

[0103] In one embodiment, the processor executing the computer program to implement the identification of the target in the image to be processed and the obtaining of the position of the identified target in the camera coordinate system comprises: determining whether the image to be processed includes the target; when the image to be processed includes the target, identifying the type of the target and the bounding box of the target in the image to be processed; and calculating the position of the target in the camera coordinate system according to the bounding box of the target in the image to be processed and a depth image.

[0104] In one embodiment, before the processor executing the computer program to implement the obtaining of the image to be processed, it further comprises: reconstructing a target map according to key frame data.

[0105] In one embodiment, the processor, when executing the computer program, implements the following before the scene reconstruction according to the key data to obtain the target map: acquiring a current frame image by the depth camera; matching the current frame image with a previous frame image; when the matching result of the features of the current frame image and the previous frame image meets a preset condition, taking the current frame image as a current key frame image; and saving the current key frame image information as the key frame data.

[0106] In one embodiment, the processor, when executing the computer program, implements the following after taking the current frame image as the current key frame image when the matching result of the features of the current frame image and the previous frame image meets the preset condition: calculating a target pose of the mobile device corresponding to the current key frame image; adjusting the current pose of the mobile device according to the target pose of the mobile device corresponding to the current key frame image, and continuing to acquire a next frame image by the depth camera as the current frame image, and continuing to match the current frame image with the previous frame image.

[0107] In one embodiment, the processor, when executing the computer program, implements the following: when the matching result of the features of the current frame image and the previous frame image does not meet the preset condition, searching for a key frame matching the current frame image from the stored key frame images.

[0108] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program, when executed by a processor, implements the following: acquiring a to-be-processed image; calculating a current pose of a mobile device in a target map coordinate system according to the to-be-processed image; identifying a target in the to-be-processed image, and acquiring a position of the identified target in a camera coordinate system; and calculating a position of the target in the target map coordinate system according to the current pose of the mobile device and the position of the target in the camera coordinate system.

[0109] In one embodiment, the computer program, when executed by the processor, implements the identification of the target in the to-be-processed image and the acquisition of the position of the identified target in the camera coordinate system, including: judging whether the to-be-processed image includes the target; when the to-be-processed image includes the target, identifying a type of the target and a bounding box of the target in the to-be-processed image; and calculating the position of the target in the camera coordinate system according to the bounding box of the target in the to-be-processed image and a depth image.

[0110] In one embodiment, the computer program, when executed by the processor, implements the following before the acquisition of the to-be-processed image: scene reconstruction according to key frame data to obtain a target map.

[0111] In one embodiment, the computer program, when executed by the processor, further implements the following steps: when the matching result of the features of the current frame image and the previous frame image does not satisfy the preset condition, searching for a key frame image matching the current frame image from the stored key frame images.

[0112] In one embodiment, the computer program, when executed by the processor, further implements the following steps after the current frame image is taken as the current key frame image when the matching result of the features of the current frame image and the previous frame image satisfies the preset condition: calculating a target pose of the self-moving device corresponding to the current key frame image; adjusting the current pose of the self-moving device according to the target pose of the self-moving device corresponding to the current key frame image, and continuing to capture the next frame image as the current frame image by the depth camera and match the current frame image with the previous frame image.

[0113] In one embodiment, the computer program, when executed by the processor, further implements the following steps: when the matching result of the features of the current frame image and the previous frame image does not satisfy the preset condition, searching for a key frame image matching the current frame image from the stored key frame images.

[0114] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps: obtaining a to-be-processed image; calculating a current pose of a self-moving device in a target map coordinate system according to the to-be-processed image; identifying a target in the to-be-processed image and obtaining a position of the identified target in a camera coordinate system; and calculating a position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system.

[0115] In one embodiment, the computer program, when executed by the processor, further implements the following steps before obtaining the to-be-processed image: reconstructing a scene according to key frame data to obtain a target map.

[0116] In one embodiment, the computer program, when executed by the processor, further implements the following steps before reconstructing a scene according to key data to obtain a target map: capturing a current frame image by a depth camera; matching the current frame image with a previous frame image; taking the current frame image as a current key frame image when a matching result of features of the current frame image and the previous frame image satisfies a preset condition; and saving information of the current key frame image as key frame data.

[0117] In one embodiment, when the matching result of the features of the current frame image and the previous frame image meets the preset condition, the computer program executed by the processor further implements the following steps: calculating a target pose of the mobile device corresponding to the current key frame image; adjusting the current pose of the mobile device according to the target pose of the mobile device corresponding to the current key frame image, and continuing to match the current frame image and the previous frame image by collecting the next frame image as the current frame image by the depth camera.

[0118] In one embodiment, when the matching result of the features of the current frame image and the previous frame image does not meet the preset condition, the computer program executed by the processor further implements the following steps: searching for a key frame matching the current frame image from the stored key frame images.

[0119] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to a memory, database or other medium in the embodiments provided by the present application can include at least one of a non-volatile and volatile memory. The non-volatile memory can include a read-only memory (ROM), a magnetic tape, a floppy disk, a flash memory, an optical storage, a high-density embedded non-volatile memory, a resistive memory (ReRAM), a magnetoresistive random access memory (MRAM), a ferroelectric memory (FRAM), a phase change memory (PCM), a graphene memory, etc. The volatile memory can include a random access memory (RAM) or an external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), etc. The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a blockchain, etc., but is not limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0120] The technical features of the above embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not contradict each other, they should be considered to be within the scope of the present disclosure.

[0121] The above embodiments only express several implementation manners of the present application, and the description is specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A target localization method, characterized in that the method The method comprises: obtaining a to-be-processed image; calculating a current pose of a self-moving device in a target map coordinate system according to the to-be-processed image; identifying a target in the to-be-processed image and obtaining a position of the identified target in a camera coordinate system; calculating a position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system; the identification of the target in the to-be-processed image and the obtaining of the position of the identified target in the camera coordinate system comprise: determining whether the to-be-processed image includes the target; when the to-be-processed image includes the target, identifying a type of the target and a bounding box of the target in the to-be-processed image; calculating the position of the target in the camera coordinate system according to the bounding box of the target in the to-be-processed image and a depth image; the calculation of the position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system comprises: calculating a conversion relationship between an origin of the camera coordinate system and the current pose according to the current pose; calculating the position of the target in the target map coordinate system according to the conversion relationship; before the obtaining of the to-be-processed image, the method further comprises: reconstructing a target map according to key frame data; before the reconstruction of the target map according to the key frame data, the method comprises: capturing a current frame image by using a depth camera; matching the current frame image with a previous frame image; when a matching result of features of the current frame image and the previous frame image meets a preset condition, taking the current frame image as a current key frame image; saving current key frame image information as the key frame data; after the taking of the current frame image as the current key frame image when the matching result of the features of the current frame image and the previous frame image meets the preset condition, the method further comprises: calculating a target pose of the self-moving device corresponding to the current key frame image; adjusting the current pose of the self-moving device according to the target pose of the self-moving device corresponding to the current key frame image, and capturing a next frame image by using the depth camera as a current frame image, and continuing to match the current frame image with a previous frame image.

2. The method of claim 1, wherein, The method further comprises: when the matching result of the features of the current frame image and the previous frame image does not meet the preset condition, searching for a key frame matching the current frame image from the stored key frame images.

3. A target positioning device, characterized by The device comprises: an image obtaining module configured to obtain a to-be-processed image; a pose calculating module configured to calculate a current pose of a self-moving device in a target map coordinate system according to the to-be-processed image; a position calculating module configured to identify a target in the to-be-processed image and obtain a position of the identified target in a camera coordinate system; a target position calculating module configured to calculate a position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system; The target position calculation module is further configured to calculate a conversion relationship between an origin in the camera coordinate system and the current pose according to the current pose; and calculate the position of the target in the target map according to the conversion relationship. The position calculation module further comprises: A judging unit configured to judge whether the target is included in the to-be-processed image; A bounding box judging unit configured to, when the target is included in the to-be-processed image, identify a type of the target and a bounding box of the target in the to-be-processed image; A camera coordinate system calculation unit configured to calculate a position of the target in the camera coordinate system according to the bounding box of the target in the to-be-processed image and the depth image; The target positioning apparatus further comprises a map reconstruction module. The map reconstruction module is configured to reconstruct a scene according to key frame data to obtain a target map. The map reconstruction module comprises: An image acquisition unit configured to acquire a current frame image by using a depth camera; An image matching unit configured to match the current frame image with a previous frame image; A key frame calculation unit configured to, when a matching result of features of the current frame image and the previous frame image meets a preset condition, take the current frame image as a current key frame image; A key frame information saving unit configured to save current key frame image information as the key frame data; A target pose calculation unit configured to calculate a target pose of a self-moving device corresponding to the current key frame image; A key frame matching unit configured to adjust a current pose of the self-moving device according to the target pose of the self-moving device corresponding to the current key frame image, and continue to match the current frame image with a previous frame image by acquiring a next frame image as a current frame image by using the depth camera.

4. The apparatus of claim 3, wherein, The map reconstruction module further comprises: A key frame searching unit configured to, when the matching result of features of the current frame image and the previous frame image does not meet the preset condition, search for a key frame matching the current frame image from stored key frame images.

5. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor, when executing the computer program, is configured to acquire a to-be-processed image; calculate a current pose of a self-moving device in a target map coordinate system according to the to-be-processed image; identify a target in the to-be-processed image and acquire a position of the identified target in a camera coordinate system; and calculate a position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system. The identifying the target in the image to be processed and obtaining the position of the identified target in the camera coordinate system comprises: judging whether the target is included in the image to be processed; when the target is included in the image to be processed, identifying the type of the target and the bounding box of the target in the image to be processed; and calculating the position of the target in the camera coordinate system according to the bounding box of the target in the image to be processed and the depth image. The calculating the position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system comprises: calculating the conversion relationship between the origin of the camera coordinate system and the current pose according to the current pose; and calculating the position of the target in the target map coordinate system according to the conversion relationship. Before the obtaining the image to be processed, the method further comprises: reconstructing a target map according to key frame data. Before the reconstructing the target map according to the key frame data, the method comprises: collecting a current frame image by using a depth camera; matching the current frame image with a previous frame image; when the matching result of the features of the current frame image and the previous frame image meets a preset condition, taking the current frame image as a current key frame image; saving the current key frame image information as the key frame data; and when the matching result of the features of the current frame image and the previous frame image meets the preset condition, taking the current frame image as the current key frame image, the method further comprises: calculating the target pose of the self-moving device corresponding to the current key frame image; adjusting the current pose of the self-moving device according to the target pose of the self-moving device corresponding to the current key frame image, and collecting a next frame image as the current frame image by using the depth camera, and continuing to match the current frame image with the previous frame image.

6. The computer device of claim 5, wherein, When the matching result of the features of the current frame image and the previous frame image does not meet the preset condition, the processor executing the computer program further implements searching for a key frame matching the current frame image from the stored key frame images.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the following steps: obtaining an image to be processed; calculating the current pose of a self-moving device in a target map coordinate system according to the image to be processed; identifying a target in the image to be processed and obtaining the position of the identified target in a camera coordinate system; and calculating the position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system. The identifying the target in the to-be-processed image and obtaining the position of the identified target in a camera coordinate system comprises: judging whether the target is included in the to-be-processed image; when the target is included in the to-be-processed image, identifying the type of the target and the bounding box of the target in the to-be-processed image; and calculating the position of the target in the camera coordinate system according to the bounding box of the target in the to-be-processed image and a depth image; the calculating the position of the target in a target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system comprises: calculating the conversion relationship between the origin of the camera coordinate system and the current pose according to the current pose; and calculating the position of the target in the target map coordinate system according to the conversion relationship; before the obtaining the to-be-processed image, the method further comprises: reconstructing a target map according to key frame data; before the reconstructing the target map according to the key frame data, the method comprises: collecting a current frame image by using a depth camera; matching the current frame image with a previous frame image; when the matching result of the features of the current frame image and the previous frame image meets a preset condition, taking the current frame image as a current key frame image; saving the current key frame image information as the key frame data; and when the matching result of the features of the current frame image and the previous frame image meets the preset condition, taking the current frame image as the current key frame image, the method further comprises: calculating the target pose of the self-moving device corresponding to the current key frame image; adjusting the current pose of the self-moving device according to the target pose of the self-moving device corresponding to the current key frame image, and collecting a next frame image as the current frame image by using the depth camera, and continuing to match the current frame image with the previous frame image.

8. The computer-readable storage medium of claim 7, wherein, When the computer program is executed by the processor, the computer program further implements, when the matching result of the features of the current frame image and the previous frame image does not meet the preset condition, searching for a key frame matching the current frame image from the stored key frame images.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the following steps: obtaining a to-be-processed image; calculating the current pose of a self-moving device in a target map coordinate system according to the to-be-processed image; identifying a target in the to-be-processed image and obtaining the position of the identified target in a camera coordinate system; and calculating the position of the target in the target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system. The identifying the target in the image to be processed and obtaining the position of the identified target in a camera coordinate system comprises: judging whether the target is included in the image to be processed; when the target is included in the image to be processed, identifying the type of the target and the bounding box of the target in the image to be processed; and calculating the position of the target in the camera coordinate system according to the bounding box of the target in the image to be processed and a depth image. The calculating the position of the target in a target map coordinate system according to the current pose of the self-moving device and the position of the target in the camera coordinate system comprises: calculating the conversion relationship between the origin of the camera coordinate system and the current pose according to the current pose; and calculating the position of the target in the target map coordinate system according to the conversion relationship. Before the obtaining the image to be processed, the method further comprises: reconstructing a target map according to key frame data. Before the reconstructing the target map according to the key frame data, the method comprises: collecting a current frame image by a depth camera; matching the current frame image with a previous frame image; when the matching result of the features of the current frame image and the previous frame image meets a preset condition, taking the current frame image as a current key frame image; saving the current key frame image information as the key frame data; when the matching result of the features of the current frame image and the previous frame image meets the preset condition, taking the current frame image as the current key frame image, the method further comprises: calculating the target pose of the self-moving device corresponding to the current key frame image; adjusting the current pose of the self-moving device according to the target pose of the self-moving device corresponding to the current key frame image, and collecting a next frame image by the depth camera as the current frame image, and continuing to match the current frame image with the previous frame image.

10. The computer program product of claim 9, wherein, When the matching result of the features of the current frame image and the previous frame image does not meet the preset condition, the computer program is further executed by the processor to find a key frame matching the current frame image from the stored key frame images.

Images