Robot data processing method, device and equipment and computer readable storage medium

Abstract

The invention provides a data processing method, device and equipment of a robot and a computer readable storage medium. The method comprises the following steps: acquiring actual image features detected by a robot during positioning, and acquiring map image features corresponding to positioning in a navigation map of the robot; when the matching degree between the actual image features and the map image features is smaller than a first threshold value, obtaining a positioned image and the positioning pose of the robot; performing association processing based on the image and the pose to obtain a key frame fragment; and updating the navigation map based on the key frame fragment to obtain an updated navigation map. According to the method and the device, the navigation map can be dynamically updated by using the change information of the actual image features, so that the timeliness and the accuracy of the navigation map are ensured, and the robot can stably run for a long time.

Description

Technical Field

[0001] The present application relates to navigation technology, and in particular to a data processing method, device, equipment and computer-readable storage medium for a robot. Background Art

[0002] In navigation technology, the navigation map is updated by real-time positioning feedback, which utilizes the association between the actual image features obtained by the robot during operation and the map image features stored in the navigation map. This update method can ensure the timeliness and accuracy of the navigation map, realize the continuous update of the navigation map, thus greatly optimizing the navigation performance and operation efficiency of the robot, providing a more stable and durable operation environment for the robot, and ensuring the stability and reliability of long-term operation.

[0003] In the related technology, when updating the navigation map by real-time positioning feedback, as long as the change information of the actual image features is obtained, the navigation map is dynamically updated. Such frequent updates will increase the computational burden and resource consumption; and the change information processed is a single key frame, with insufficient ability to capture continuous dynamic changes; when deleting redundant information in the navigation map, it is based on the relationship between the new key frame used for updating the navigation map and the old key frame corresponding to the new key frame in the navigation map before updating, resulting in poor comprehensiveness in capturing redundant information in the navigation map. Summary of the Invention

[0004] Embodiments of the present application provide a data processing method, device, equipment and computer-readable storage medium for a robot, which can dynamically update the navigation map using the change information of the actual image features, ensure the timeliness and accuracy of the navigation map, and enable the robot to operate stably for a long time.

[0005] The technical solution of the embodiments of the present application is implemented as follows:

[0006] Embodiments of the present application provide a data processing method for a robot, the method comprising:

[0007] Obtain the actual image features detected by the robot during positioning, and obtain the map image features corresponding to the positioning in the navigation map of the robot;

[0008] When the matching degree between the actual image features and the map image features is less than a first threshold, obtain a plurality of images at the positioning and a plurality of poses of the robot at the positioning;

[0009] Perform association processing based on the plurality of images and the plurality of poses to obtain a key frame segment;

[0010] Update the navigation map based on the key frame segment to obtain an updated navigation map.

[0011] An embodiment of the present application provides a data processing device for a robot, including:

[0012] An acquisition module, configured to acquire the actual image features detected by the robot during positioning, and acquire the map image features corresponding to the positioning in the navigation map of the robot; when the matching degree between the actual image features and the map image features is less than a first threshold, acquire a plurality of images at the positioning and a plurality of poses of the robot at the positioning;

[0013] An association module, configured to perform association processing based on the plurality of images and the plurality of poses to obtain a key frame segment;

[0014] An update module, configured to update the navigation map based on the key frame segment to obtain an updated navigation map.

[0015] In the above solution, the association module is further configured to screen out a plurality of static valid images from the plurality of images, and obtain valid poses corresponding to each of the static valid images from the plurality of poses; for each of the static valid images, extract feature points of the static valid image, and form a key frame corresponding to the static valid image by combining the feature points of the static valid image with the valid pose corresponding to the static valid image; perform association processing based on the key frames corresponding to each of the static valid images to obtain the key frame segment.

[0016] In the above solution, the association module is further configured to group a plurality of the key frames to obtain at least one key frame combination, where a plurality of key frames included in the key frame combination have the same plurality of feature points; for each of the key frame combinations, associate the key frames included in the key frame combination in chronological order to obtain a key frame segment corresponding to the key frame combination.

[0017] In the above solution, the association module is further configured to perform target recognition processing on each of the images to obtain the occupancy area of the semantic dynamic object in each of the images; obtain a plurality of images with the occupancy area less than a second threshold from the plurality of images as static images; for each of the static images, obtain a first difference between the static image and the historical static image corresponding to the static image; use the static image with the first difference greater than a third threshold as the static valid image, where the acquisition time of the historical static image is earlier than the acquisition time of the static image.

[0018] In the above solution, the association module is further configured to obtain a second difference between the pose corresponding to the static image and the pose corresponding to the historical static image; when the second difference is less than a fourth threshold, compare the static image with the historical static image to obtain the first difference.

[0019] In the above solution, the update module is further configured to add the key frame segment to the navigation map to obtain a first intermediate navigation map; delete redundant map points corresponding to the key frame segment from the first intermediate navigation map to obtain a second intermediate navigation map; optimize the key frame segment in the second intermediate navigation map based on the navigation map to obtain an updated navigation map.

[0020] In the above solution, the update module is further configured to match the pose corresponding to the first key frame in the key frame segment with the poses corresponding to multiple map points in the navigation map to obtain multiple first matching values; use the map point corresponding to the largest first matching value among the multiple first matching values as the first matching map point; generate new map points corresponding to each key frame in the key frame segment, and generate new map edges based on the association relationship between the key frames in the key frame segment; add the new map points and the new map edges to the first matching map point in the navigation map to obtain the first intermediate navigation map.

[0021] In the above solution, the update module is further configured to obtain a third difference between the pose corresponding to each map point in the first intermediate navigation map and the pose corresponding to the key frame segment; use the map points corresponding to the third differences less than a fifth threshold as the map points to be processed for redundancy; obtain a fourth difference between the map points to be processed for redundancy and historical key frames, where the historical key frames are obtained by the robot at the positions corresponding to the map points to be processed for redundancy; use the map points to be processed for redundancy corresponding to the fourth differences greater than a sixth threshold as the redundant map points; the step of deleting the redundant map points corresponding to the key frame segment from the first intermediate navigation map to obtain a second intermediate navigation map includes, when the number of redundant map points is greater than a seventh threshold, deleting the redundant map points in the first intermediate navigation map to obtain the second intermediate navigation map.

[0022] In the above solution, the update module is further configured to obtain, from multiple map points in the navigation map, second matching map points that match the key frame segment; align the navigation map and the second intermediate navigation map based on the second matching map points and the key frames in the key frame segment that correspond to the second matching map points, to obtain a third intermediate navigation map; perform feature matching processing on the image features corresponding to the second matching map points and the image features of the key frames in the key frame segment that correspond to the second matching map points, to obtain a feature matching result; and optimize the key frame segment in the second intermediate navigation map based on the feature matching result, to obtain the updated navigation map.

[0023] An embodiment of the present application provides an electronic device, which includes:

[0024] a memory, configured to store computer-executable instructions or computer programs;

[0025] a processor, configured to implement the data processing method of the robot provided by the embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0026] An embodiment of the present application provides a computer-readable storage medium, storing a computer program or computer-executable instructions, which are configured to implement the data processing method of the robot provided by the embodiment of the present application when being executed by a processor.

[0027] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions, where when the computer program or computer-executable instructions are executed by a processor, the data processing method of the robot provided by the embodiment of the present application is implemented.

[0028] The embodiments of the present application have the following beneficial effects: obtaining the actual image features detected by the robot during positioning, and obtaining the map image features corresponding to the positioning in the navigation map of the robot; when the matching degree between the actual image features and the map image features is less than the first threshold, obtaining multiple images of the positioning and multiple poses of the robot at the positioning; when the actual image features do not match the map image features due to changes in the actual environment, it is possible to obtain multiple images of the positioning related to the update of the navigation map and multiple poses of the robot at the positioning, determine the timing of updating the navigation map, ensure that the navigation map can be updated in a timely manner, and improve the accuracy of navigation. Based on the multiple images and the multiple poses, correlation processing is performed to obtain a key frame segment; the key frame segment obtained by the correlation processing can accurately capture and reflect the continuous dynamic change information of the actual environment, and can provide more comprehensive and real-time environmental data support for the navigation and positioning of the robot. Based on the key frame segment, the navigation map is updated to obtain an updated navigation map. It is possible to use the key frame segment to update the navigation map and maintain the timeliness and accuracy of the navigation map. Through the embodiments of the present application, it is possible to dynamically update the navigation map using the change information of the actual image features, ensure the timeliness and accuracy of the navigation map, and enable the robot to operate stably for a long time. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic structural diagram of the data processing system 100 architecture of the robot provided by the embodiments of the present application;

[0030] Figure 2 is a schematic structural diagram of the data processing device of the robot provided by the embodiments of the present application;

[0031] Figures 3A - 3H is a schematic flowchart of the data processing method of the robot provided by the embodiments of the present application;

[0032] Figure 4 is a schematic framework diagram of the visual map automatic update method during the operation process of the robot provided by the embodiments of the present application;

[0033] Figure 5 is a schematic diagram of the environmental information acquisition module provided by the embodiments of the present application;

[0034] Figure 6 is a schematic diagram of the map update module provided by the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] In order to make the objectives, technical solutions, and advantages of this application clearer, the following will further describe this application in detail with reference to the accompanying drawings. The described embodiments should not be regarded as limitations of this application. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

[0036] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0037] In the following description, the terms "first / second / third" are only used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged in a specific order or sequence when permitted, so that the embodiments of this application described here can be implemented in an order other than that illustrated or described here.

[0038] In the embodiments of this application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the function of that module or unit.

[0039] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used in the embodiments of this application are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0040] In the embodiments of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations during actual application, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope authorized by laws, regulations, and the personal information subject.

[0041] Before further elaborating on the embodiments of this application, the nouns and terms involved in the embodiments of this application are explained. The nouns and terms involved in the embodiments of this application are subject to the following explanations.

[0042] (1) Pose: It refers to the position and orientation of the robot. The position represents the coordinates of the center or reference point of the robot in three-dimensional space. Usually, three real numbers are used to represent the three-dimensional coordinates of a point, and these three real numbers correspond to the distances on the X-axis, Y-axis, and Z-axis respectively. In three-dimensional space, the position of a robot can be accurately located through a three-dimensional coordinate system. The orientation represents the direction or pose of the robot in three-dimensional space. The ways to describe the orientation include rotation matrix, Euler angles, and quaternions, etc.

[0043] (2) Semantic dynamic object: In the navigation map, it is a dynamic entity that is given specific semantic information according to the perception, understanding, and interaction requirements of the robot. These entities can be objects, people, or events around the robot, etc. The robot senses the surrounding environment through sensors such as cameras and lidar, and identifies semantic dynamic objects, such as pedestrians, vehicles, obstacles, etc.

[0044] (3) Topological map: The topological map maintains the correctness of the relative position relationship between points and lines, but does not necessarily maintain the accuracy of the graphic shape, area, distance, and direction. It uses nodes and edges to represent the important position points in the environment and the connection relationships between them. It is a highly abstract map representation that ignores specific shape, area, and distance information, and only focuses on the relative position relationship between points and lines. Nodes are the basic elements in the topological map, representing important position points in the environment, such as corners, doorways, elevators, stairs and other entities. Edges represent the connection relationships between nodes, such as corridors, passages, etc. In the topological map, edges are usually undirected, but direction information can also be added according to needs.

[0045] (4) Triangulation: Based on the principles of visual geometry, it uses feature points or corresponding points under multiple perspectives for measurement and analysis, and calculates the positions of these feature points in three-dimensional space.

[0046] (5) Map point: It refers to a marker used to represent a specific position or geographical coordinates on the navigation map. For example, it can be a location with actual geographical coordinates such as longitude and latitude, such as the intersection of a city, a mountain range, and a river. It can also be a symbol marked at a specific position on the navigation map, such as a restaurant, a hospital, a shopping mall, etc.

[0047] (6) Redundant map point: It refers to a map point in the navigation map where there are significant differences between the recorded image or pose information and the actually collected image or pose information. The existence of redundant map points will cause inaccurate positioning or errors of the robot at these redundant map points. Since each key frame in the key frame segment corresponds to a map point, the redundant map point corresponding to the key frame segment refers to a key frame in the key frame segment that has the same positioning as the redundant map point.

[0048] In navigation technology, the update of the navigation map through real-time positioning feedback is carried out by using the association between the actual image features obtained by the robot during operation and the map image features stored in the navigation map. This update method can ensure the timeliness and accuracy of the navigation map, realize the continuous update of the navigation map, thus greatly optimizing the navigation performance and operation efficiency of the robot, providing a more stable and durable operation environment for the robot, and ensuring the stability and reliability of long-term operation.

[0049] In the related art, when updating the navigation map through real-time positioning feedback, as long as the change information of the actual image features is obtained, the navigation map is dynamically updated. Such frequent updates will increase the computational burden and resource consumption; and the change information processed is a single key frame, lacking the ability to capture continuous dynamic changes; when deleting redundant information in the navigation map, it is based on the relationship between the new key frame used for updating the navigation map and the old key frame corresponding to the new key frame in the navigation map before updating, resulting in poor comprehensiveness in capturing redundant information in the navigation map.

[0050] The embodiments of the present application provide a data processing method, device, equipment and computer-readable storage medium for a robot, which can dynamically update the navigation map using the change information of the actual image features, synchronously delete redundant information in the navigation map, ensure the timeliness and accuracy of the navigation map, and enable the robot to operate stably for a long time. The following describes the exemplary applications of the electronic equipment provided by the embodiments of the present application. The equipment provided by the embodiments of the present application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, vehicle terminals, etc., or can also be implemented as a server. Next, the exemplary applications when the equipment is implemented as a terminal will be described.

[0051] See Figure 1 , Figure 1 FIG. is a schematic structural diagram of the data processing system 100 architecture of the robot provided by the embodiments of the present application. To support a data processing application of a robot, the terminal 400 is connected to the server 200 through the network 300, and the server 200 uploads data to the database 500. The network 300 can be a wide area network, a local area network, or a combination of the two.

[0052] The terminal 400 acquires the actual image features detected by the robot during positioning and acquires the map image features corresponding to the positioning in the navigation map of the robot; the terminal 400 uploads the acquired actual image features and map image features to the server 200, and the server 200 matches the actual image features and the map image features. When the matching degree between the actual image features and the map image features is less than the first threshold, the server 200 sends an acquisition instruction to the terminal 400, and the terminal 400 acquires a plurality of images of the positioning and a plurality of poses of the robot at the positioning, and uploads the acquired information to the server 200. The server 200 performs an association process based on the plurality of images and the plurality of poses to obtain a key frame segment; and based on the key frame segment, updates the navigation map to obtain an updated navigation map. The server 200 deploys the updated navigation map in the terminal 400 for use in the navigation of the robot.

[0053] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected through wired or wireless communication methods, which are not limited in the embodiments of the present application.

[0054] See Figure 2 , Figure 2 is a schematic structural diagram of a data processing device of a robot provided by an embodiment of the present application. Figure 2 The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. Each component in the terminal 400 is coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. The bus system 440 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in Figure 2 all the buses are labeled as the bus system 440.

[0055] The processor 410 may be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a Digital Signal Processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0056] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0057] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0058] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0059] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0060] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0061] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0062] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0063] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0064] In some embodiments, the device provided by the embodiments of the present application may be implemented in software. Figure 2 FIG. Figure 2 shows a data processing device 455 of a robot stored in a memory 450, which may be software in the form of a program and a plug-in, etc., including the following software modules: an acquisition module 4551, an association module 4552, and an update module 4553. These modules are logical, and thus can be arbitrarily combined or further split according to the functions to be implemented. The functions of each module will be described below.

[0065] In other embodiments, the device provided by the embodiments of the present application may be implemented in hardware. As an example, the device provided by the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method of the robot provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may employ one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0066] In some embodiments, a terminal or a server may implement the data processing method of the robot provided by the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions may be commands at the microprogram level, machine instructions, or software instructions. The computer program may be a native program or a software module in an operating system; it may be a native application (APPlication, APP), that is, a program that needs to be installed in an operating system to run, or it may be a small program that can be embedded in any APP, that is, a program that only needs to be downloaded to a browser environment to run. In short, the above computer-executable instructions may be instructions in any form, and the above computer program may be an application program, a module, or a plug-in in any form.

[0067] The data processing method of the robot provided by the embodiments of the present application will be described in combination with the exemplary applications and implementations of the terminal provided by the embodiments of the present application.

[0068] Next, the data processing method of the robot provided by the embodiments of the present application will be described. As mentioned above, the electronic device implementing the data processing method of the robot in the embodiments of the present application can be a terminal, a server, or a combination of both. Therefore, the execution subject of each step will not be repeated hereinafter.

[0069] Refer to Figure 3A , Figure 3A which is a schematic flowchart of the data processing method of the robot provided by the embodiments of the present application, and will be described in conjunction with Figure 3A the steps shown. Figure 3A The subject of the step is the terminal.

[0070] In step 101, obtain the actual image features detected during the positioning of the robot, and obtain the map image features corresponding to the positioning in the navigation map of the robot.

[0071] As an example, when the robot moves to a certain position, it will use its camera or other sensors such as lidar, infrared sensors, etc. to capture the actual image information of the surrounding environment based on the positioning of the robot. By extracting the image features from the actual image information, the actual image features are obtained. Based on the positioning of the robot, the map image information corresponding to the positioning in the navigation map is obtained. By extracting the image features from the map image information, the map image features are obtained. The features in the actual image features and the map image features can be the edges, corner points, color patterns, textures, etc. of objects.

[0072] In step 102, when the matching degree between the actual image features and the map image features is less than the first threshold, obtain multiple images of the positioning and multiple poses of the robot at the positioning.

[0073] As an example, it is necessary to match the actual image features and the map image features to determine whether there are differences between the actual image features and the map image features. When the matching degree between the actual image features and the map image features is less than the first threshold, it means that the map image features recorded in the navigation map are too different from the actual image features collected by the robot. At this time, the navigation map needs to be updated. To update the navigation map, it is necessary to obtain multiple images collected by the robot during the positioning, as well as the poses. The pose is the position and direction of the robot. The position represents the coordinates of the center or reference point of the robot in three-dimensional space. Usually, three real numbers are used to represent the three-dimensional coordinates of a point, and these three real numbers correspond to the distances on the X-axis, Y-axis, and Z-axis respectively. In three-dimensional space, the position of a robot can be accurately located through a three-dimensional coordinate system. The direction represents the orientation or pose of the robot in three-dimensional space. The ways to describe the direction include rotation matrix, Euler angles, and quaternions, etc. The value of the first threshold is related to the requirements of actual applications and is not limited here.

[0074] In step 103, association processing is performed based on the multiple images and the multiple poses to obtain a key frame segment.

[0075] In some embodiments, referring to Figure 3B , Figure 3A the steps shown in 103 can be implemented by the following steps 1031 to 1034, which will be described below in conjunction with Figure 3B for illustration.

[0076] In step 1031, a plurality of static valid images are screened out from the multiple images, and valid poses corresponding to each of the static valid images are obtained from the multiple poses.

[0077] In some embodiments, referring to Figure 3C , Figure 3B the steps shown in 1031 can be implemented by the following steps 1031A to 1031D, which will be described below in conjunction with Figure 3C for illustration.

[0078] In step 1031A, target recognition processing is performed on each of the images to obtain the occupied area of the semantic dynamic object in each of the images.

[0079] As an example, through target recognition processing, the recognized semantic dynamic object in the image, after target recognition processing, the position and size information of the semantic dynamic object in each image will be obtained. These information are usually represented in the form of a bounding box, and the bounding box encloses the area of the target object, that is, the semantic dynamic object in the image. Calculating the occupied area is based on the information of these bounding boxes to calculate the area occupied by the recognized semantic dynamic object in the image. First, calculate the area of the image, and the area of the image can be obtained by the rectangle area formula, that is, the area is equal to the image width multiplied by the image height. If the bounding box corresponding to the semantic dynamic object is a rectangle, then use the rectangle area formula, that is, the area is equal to the bounding box width multiplied by the bounding box height to obtain the area of the semantic dynamic object; if the bounding box corresponding to the semantic dynamic object is a non-rectangular bounding box, then more complex geometric methods such as the shoelace formula or the vector cross product method can be used to obtain the area of the semantic dynamic object. By comparing the area of the semantic dynamic object with the area of the image, the occupied area of the semantic dynamic object in the image is obtained, and the specific calculation formula is shown in the following formula (1).

[0080]

[0081] Among them, S is the occupied area of the semantic dynamic object in the image, S1 is the area of the semantic dynamic object, and S2 is the area of the image.

[0082] In step 1031B, a plurality of images with the occupied area less than the second threshold are obtained from the multiple images as static images.

[0083] As an example, after obtaining the occupied area of the semantic dynamic object in all images, it is necessary to filter the images according to the size of the occupied area of the semantic dynamic object in the images to obtain static images. The value of the second threshold is related to the requirements of the actual application and is not limited here. Selecting the image with the occupied area smaller than the second threshold as the static image is because the semantic dynamic object in the image will affect the positioning accuracy. An image with an overly large occupied area of the semantic dynamic object will cause problems in the positioning of the robot. Therefore, when updating the navigation map, the image with a smaller occupied area of the semantic dynamic object, that is, the static image, is used.

[0084] In step 1031C, for each of the static images, obtain a first difference between the static image and the historical static image corresponding to the static image.

[0085] As an example, when the robot moves slowly or the image feature information in the environment where the robot operates changes little, that is, when the difference between the static image and the historical static image is small, there will be no major problem in the positioning of the navigation map. To save computing resources, the operation of updating the navigation map can be omitted at this time. Therefore, when the difference between the obtained static image and the historical static image is too large, the navigation map is updated.

[0086] In some embodiments, Figure 3C The shown step 1031C can be implemented by the following technical solution: obtain a second difference between the pose corresponding to the static image and the pose corresponding to the historical static image; when the second difference is less than a fourth threshold, compare the static image and the historical static image to obtain the first difference.

[0087] As an example, before determining the first difference between the static image and the historical static image corresponding to the static image, it is necessary to determine the second difference between the pose corresponding to the static image and the pose corresponding to the historical static image, that is, to determine whether the static image and the historical static image are images obtained under the same pose. When the second difference between the pose corresponding to the static image and the pose corresponding to the historical static image is less than the fourth threshold, it is determined that the static image and the historical static image are images obtained under the same pose. The fourth threshold is related to the actual requirements and is not limited here.

[0088] As an example, determining the second difference between the posture corresponding to the static image and the posture corresponding to the historical static image can be achieved by the following technical solution: first, obtaining the first rotation matrix and the first translation vector of the posture corresponding to the static image, as well as the second rotation matrix and the second translation vector of the posture corresponding to the historical static image; the posture matrix is a 4x4 homogeneous transformation matrix that contains rotation and translation information. The first rotation matrix R1 and the first translation vector t1 are obtained by: obtaining the matrix of the posture corresponding to the static image Where R1 is a 3×3 rotation matrix, t1 is a 3×1 translation vector, 0 is a 3×1 zero vector, and 1 is a scalar. The first rotation matrix R1 is the 3x3 submatrix of the upper left corner of T1, and the first translation vector t1 is the 3×1 column vector to the right of T1, excluding the last row. The second rotation matrix R2 and the second translation vector t2 are obtained in the same way as the first rotation matrix R1 and the first translation vector t1, and will not be repeated here. Based on the first translation vector and the second translation vector, the translation difference d is determined; the translation difference d can be obtained by calculating the Euclidean distance between t1 and t2. The specific calculation method is shown in the following formula (2).

[0089]

[0090] Where d is the translation difference, the first translation vector t1 = {x1, y1, z1}, and the second translation vector t2 = {x2, y2, z2}. The first rotation matrix and the second rotation matrix are mapped using inverse trigonometric functions to obtain the rotation difference. The rotation difference is calculated as shown in the following formula (3).

[0091] θ=2·arccos(|R1·R2|) (3)

[0092] Where θ is the rotation difference, R1 is the first rotation matrix, R2 is the second rotation matrix, and the inverse trigonometric function is arccos (the inverse cosine function). Based on the translation difference, the translation weight, the rotation difference, and the rotation weight, a second difference is determined. The specific method for determining the second difference is shown in the following formula (4).

[0093] E=W d d+W θ ·θ (4)

[0094] Among them, E is the second difference, W d is the translation difference weight, d is the translation difference, W θ is the translation difference weight, and θ is the rotation difference. d is the translation difference weight and W θ The value of the translation difference weight depends on the actual requirements and is not limited here. The smaller the second difference E, the closer the pose matrices T1 and T2 are.

[0095] As an example, when the second difference is less than the fourth threshold, the static image and the historical static image are compared to obtain the first difference, which can be achieved by the following technical solution: Image feature extraction is performed on the static image and the historical static image to obtain static image features and historical static image features. The feature extraction method includes obtaining the gray value distribution by calculating the histogram of the image to obtain the image brightness feature; using methods such as structural similarity to extract the structural features of the image, such as brightness, contrast, and structural details; using methods such as gray-level co-occurrence matrix, filters, and local binary pattern to extract the texture features of the image; and obtaining the shape features through mathematical methods such as moments to describe the overall or local shape of the image. Based on the static image features and the historical static image features, the similarity degree between the static image and the historical static image, that is, the first difference, is determined through a feature point matching algorithm.

[0096] Through the embodiments of the present application, images with similar poses but large content differences can be determined, improving the image comparison efficiency. The subtle differences between images can be accurately captured, improving the matching accuracy. When the difference between the poses meets the conditions, the differences between the images are compared, which can reduce unnecessary image comparison operations and save computing resources and time costs.

[0097] In step 1031D, the static image with the first difference greater than the third threshold is used as the static valid image.

[0098] As an example, the acquisition time of the historical static image is earlier than the acquisition time of the static image. When the first difference is greater than the third threshold, it indicates that the difference between the static image and the historical static image is large. The value of the third threshold is related to the actual requirements and is not limited here.

[0099] Through the embodiments of the present application, target recognition can be performed on each image to determine the occupied area of the semantic dynamic object, and the image with the occupied area less than the second threshold is screened out as the static image, effectively reducing the interference of the semantic dynamic object on the update of the navigation map. By comparing the first difference between the static image and its historical static image, when the difference is greater than the third threshold, it is regarded as a static valid image, which can accurately identify and extract the static image with significant scene changes, providing a high-quality data basis for subsequent image analysis and processing, while reducing the computational complexity and improving the processing efficiency.

[0100] Continue to refer to Figure 3B In step 1032, for each of the static valid images, feature points of the static valid image are extracted.

[0101] As an example, the method for extracting feature points includes obtaining the gray value distribution by calculating the histogram of the static valid image to obtain the brightness feature of the static valid image; extracting the structural features of the static valid image, such as brightness, contrast, and structural details, by using methods such as structural similarity; extracting the texture features of the static valid image by methods such as gray-level co-occurrence matrix, filter, and local binary pattern; and obtaining the shape features through mathematical methods such as moments to describe the overall or local shape of the static valid image. By triangulating the feature points of the static valid image, the corresponding pose of the static valid image can be obtained.

[0102] In step 1033, the feature points of the static valid image and the valid pose corresponding to the static valid image form a key frame corresponding to the static valid image.

[0103] As an example, the feature points of the static valid image are associated according to the valid pose such as the position and direction corresponding to the static valid image during shooting, that is, the coordinates and descriptors of the feature points of the static valid image and the pose matrix corresponding to the valid pose are jointly stored in a data structure to obtain the key frame of the static valid image.

[0104] In step 1034, based on the key frames corresponding to each static valid image, association processing is performed to obtain the key frame segment.

[0105] Through the embodiments of the present application, static valid images are screened out, feature points of the static valid images are extracted, and valid poses corresponding to the static valid images are obtained to form key frames, and then the key frames are associated to form key frame segments, which can efficiently integrate images and poses, accurately capture dynamic changes in the scene, and improve the pertinence and efficiency of image processing.

[0106] In some embodiments, referring to Figure 3D , Figure 3B The steps 1034 shown can be implemented through the following steps 1034A to 1034B, which will be described below in conjunction with Figure 3D for illustration.

[0107] In step 1034A, multiple key frames are grouped to obtain at least one key frame combination.

[0108] As an example, the multiple key frames included in the key frame combination have the same multiple feature points. Based on the same multiple feature points, the key frames are grouped to obtain at least one key frame combination.

[0109] In step 1034B, for each key frame combination, the key frames included in the key frame combination are associated in chronological order to obtain the key frame segment corresponding to the key frame combination.

[0110] As an example, each combination of key frames has a chronological order during acquisition. By associating multiple key frames in the chronological order of acquisition, the obtained key frame segment can represent the static image information collected by the robot within a period of time and the changes in the poses corresponding to the static image information.

[0111] Through the embodiments of the present application, multiple key frames are grouped, the key frames are combined together based on the same feature points, and these key frames are associated based on the time order to form a key frame segment, which can efficiently integrate and utilize the information in the image. It can not only accurately capture the static and dynamic changes in the scene, but also improve the efficiency and accuracy of image processing, and significantly enhance the practicability and robustness of the system.

[0112] Continue to refer to Figure 3A , in step 104, based on the key frame segment, the navigation map is updated to obtain an updated navigation map.

[0113] In some embodiments, refer to Figure 3E , Figure 3A The step 104 shown can be implemented by the following steps 1041 to 1043. The following is described in conjunction with Figure 3E for illustration.

[0114] In step 1041, the key frame segment is added to the navigation map to obtain a first intermediate navigation map.

[0115] In some embodiments, refer to Figure 3F , Figure 3E The step 1041 shown can be implemented by the following steps 1041A to 1041E. The following is described in conjunction with Figure 3F for illustration. It should be noted that the key frame segment includes multiple key frames.

[0116] In step 1041A, the pose corresponding to the first key frame in the key frame segment is matched with the poses corresponding to multiple map points in the navigation map to obtain multiple first matching values.

[0117] As an example, the step of matching the pose corresponding to the first key frame in the first key frame segment with the poses corresponding to multiple map points in the navigation map is specifically: through Figure 3C the second difference determined by the step 1031C shown, the value negatively correlated with the second difference is the first matching value. That is, the smaller the second difference, the larger the corresponding first matching value.

[0118] In step 1041B, the map point corresponding to the largest first matching value among the multiple first matching values is used as the first matching map point.

[0119] As an example, when adding a key-frame segment to a navigation map, it is necessary to find the map point that is most similar in pose to the first key frame in the key-frame segment as the first matching map point, and add the key-frame segment after the first matching map point. The larger the first matching value is, the more similar the poses of the first key frame and the map point are. Therefore, it is necessary to select the map point corresponding to the largest first matching value among multiple first matching values as the first matching map point.

[0120] In step 1041C, new map points corresponding to each key frame in the key-frame segment are generated.

[0121] In step 1041D, new map edges are generated based on the association relationships between the key frames in the key-frame segment.

[0122] As an example, the navigation map is a topological map, that is, an abstract map composed of points and edges. Therefore, it is necessary to process the key-frame segment to obtain new map points and new map edges. Each key frame in the key-frame segment is abstracted into a new map point, and the association relationship between the key-frame segments, that is, the relationship in which the key frames are grouped according to feature points and associated in chronological order, is abstracted into a new map edge. For each new map point, the corresponding key frame is stored, and for each new map edge, the two new map points it connects, that is, the key frames, are stored.

[0123] In step 1041E, the new map points and the new map edges are added to the first matching map point in the navigation map to obtain the first intermediate navigation map.

[0124] As an example, after obtaining the new map points and new map edges corresponding to the key-frame segment, the new map points and new map edges are added after the first matching map point in the navigation map to obtain the first intermediate navigation map.

[0125] Through the embodiments of the present application, the pose of the first frame of the key-frame segment is matched with the pose of the map point in the navigation map, the optimal matching point is selected as the insertion position of the key-frame segment, new map points and map edges are generated from each key frame in the key-frame segment and the association relationships between the key frames, and then integrated into the navigation map, which can dynamically update the navigation map and ensure the timeliness and accuracy of the map information. It not only enhances the integrity and coherence of the map, but also provides a more reliable environmental model for navigation, significantly improving the positioning accuracy and path planning ability of the navigation map.

[0126] Continue to refer to Figure 3E , in step 1042, redundant map points corresponding to the key-frame segment are deleted from the first intermediate navigation map to obtain the second intermediate navigation map.

[0127] In some embodiments, referring to Figure 3G , Figure 3E step 1042 shown can be implemented through the following steps 1042A to 1042E, which will be described below in conjunction with Figure 3G .

[0128] In step 1042A, obtain a third difference between the pose corresponding to each map point in the first intermediate navigation map and the pose corresponding to the key frame segment.

[0129] As an example, there are redundant map points to be processed near the key frame segment in the first intermediate navigation map. The poses of the redundant map points to be processed near the key frame segment are similar to those of the key frame segment. Therefore, it is necessary to determine the third difference between the pose corresponding to each map point in the first intermediate navigation map and the pose corresponding to the key frame segment. The method for determining the third difference is the same as that of Figure 3C step 1031C shown for obtaining the second difference, which will not be elaborated here.

[0130] In step 1042B, use the map points corresponding to the third differences less than the fifth threshold as the redundant map points to be processed.

[0131] As an example, the smaller the third difference, the more it indicates that the pose of the redundant map point to be processed corresponding to the key frame segment is similar to that of the key frame segment.

[0132] In step 1042C, obtain a fourth difference between the redundant map points to be processed and the historical key frames.

[0133] As an example, the historical key frames are obtained by the robot at the position corresponding to the redundant map points to be processed. During the operation of the robot, it is necessary to count the positioning situation of each map point. Specifically, when the robot is at the position of the redundant map points to be processed, it is necessary to determine whether the collected image and pose information, that is, the historical key frames, are consistent with the image and pose information of the redundant map points to be processed. First, it is necessary to determine the difference between the pose of the redundant map points to be processed and the pose of the historical key frames. When the difference between the pose of the redundant map points to be processed and the pose of the historical key frames is small, it can be considered that the positioning of the redundant map points to be processed and the historical key frames is the same. At this time, determine the fourth difference between the image of the redundant map points to be processed and the image of the historical key frames. The steps for determining the difference between the poses and the difference between the images are the same as those of Figure 3C step 1031C shown, which will not be elaborated here.

[0134] In step 1042D, use the redundant map points to be processed corresponding to the fourth differences greater than the sixth threshold as the redundant map points.

[0135] As an example, if the image of the redundant map point to be processed differs significantly from the image of the historical key frame, it indicates that the information provided by the redundant map point to be processed in the first intermediate navigation map is inaccurate, and there will be problems with the robot's positioning at the redundant map point to be processed. Therefore, the image of the redundant map point to be processed corresponding to when the fourth difference from the image of the historical key frame is greater than the sixth threshold is used as the redundant map point for redundancy processing. The value of the sixth threshold is related to the actual requirements and is not limited here.

[0136] In step 1042E, when the number of the redundant map points is greater than the seventh threshold, the redundant map points are deleted from the first intermediate navigation map to obtain the second intermediate navigation map.

[0137] As an example, in order to reduce the situation where frequent updates of the navigation map lead to instability of the navigation map and waste of computing resources, it is necessary to calculate the number of redundant map points, and only when the number of redundant map points exceeds a certain threshold, deletion is performed. The value of the seventh threshold is related to the actual requirements and is not limited here.

[0138] Through the embodiments of the present application, the accuracy and clarity of the navigation map can be effectively improved, redundant information in the navigation map can be accurately removed, and the expansion and positioning error of the navigation map are avoided. When the number of redundant map points exceeds the set threshold, the redundant map points are automatically deleted to ensure the simplicity and efficiency of the navigation map, providing a more accurate and reliable navigation environment for the robot, and significantly improving the stability and practicality of the navigation map.

[0139] Continue to refer to Figure 3E , in step 1043, based on the navigation map, the key frame segment in the second intermediate navigation map is optimized to obtain an updated navigation map.

[0140] Through the embodiments of the present application, after adding the key frame segment to the navigation map and removing redundant points, and then optimizing the key frame segment, the accuracy and practicality of the navigation map can be significantly improved. It not only enriches the information of the navigation map, enhances the integrity and coherence of the navigation map, but also effectively avoids the influence of redundant map points on the navigation accuracy. The optimized navigation map provides a clearer and more accurate navigation guidance for the robot, significantly improving the efficiency and safety of navigation.

[0141] In some embodiments, refer to Figure 3H , Figure 3E shown in step 1043 can be implemented through the following steps 1043A to 1043D, which will be described below in conjunction with Figure 3H for illustration.

[0142] In step 1043A, second matching map points that match the key frame segment are obtained from the multiple map points in the navigation map.

[0143] As an example, since the navigation map has been calibrated during generation, the pose of the navigation map is accurate. Since the running route and mode of the robot are relatively fixed, the change in the pose of the robot is relatively small. What changes significantly is the image collected by the robot during operation. After obtaining the key frame segment, it is necessary to correct the pose of the key frame segment. The pose of each key frame in the key frame segment is matched with the poses of multiple map points in the navigation map. The matching process is to determine the second difference through Figure 3C the step 1031C shown. The value negatively correlated with the second difference is the matching value, that is, the smaller the second difference, the larger the corresponding matching value. The map point corresponding to the largest matching value is selected from multiple matching values as the second matching map point. The larger the matching value, the more similar the pose of the key frame in the key frame segment is to the pose of the map point.

[0144] In step 1043B, based on the second matching map point and the key frame in the key frame segment corresponding to the second matching map point, the navigation map and the second intermediate navigation map are aligned to obtain a third intermediate navigation map.

[0145] As an example, the navigation map and the second intermediate navigation map are aligned because there are similar parts in the second intermediate navigation map to those in the navigation map. The second matching map point obtained through step 1043A can align the navigation map and the second intermediate navigation map. It is more accurate to perform pose correction based on the aligned third intermediate navigation map.

[0146] In step 1043C, feature matching processing is performed on the image features corresponding to the second matching map point and the image features of the key frame in the key frame segment corresponding to the second matching map point to obtain a feature matching result.

[0147] As an example, the feature matching matches the image features corresponding to the second matching map point and the image features of the key frame in the key frame segment corresponding to the second matching map point. Through feature matching, it can be determined that the image corresponding to the second matching map point is similar to the image of the key frame in the key frame segment corresponding to the second matching map point. Through feature matching, the relationship between the feature points of the image corresponding to the second matching map point and the image of the key frame in the key frame segment corresponding to the second matching map point can be obtained. The position relationship of the feature points in the space of the image can be deduced through the triangulation method.

[0148] In step 1043D, based on the feature matching result, the key frame segment in the second intermediate navigation map is optimized to obtain the updated navigation map.

[0149] As an example, when performing key-frame segment optimization, the map points in the navigation map are used as observation information, and the pose of the newly added key-frame segment and the pose of its corresponding map point are optimized using the feature matching result. When it is found that the pose of a certain key frame in the key-frame segment is too different from the pose of the map point in the navigation map, the corresponding map point in the second intermediate navigation map is deleted. The difference between poses can be determined by the second difference obtained in step 1031C shown in Figure 3C The greater the second difference, the greater the difference between the poses.

[0150] Through the embodiments of the present application, the navigation map can be aligned, and the aligned navigation map is combined with the image feature matching result to optimize the key-frame segment, further improving the accuracy and consistency of the navigation map, and enhancing the reliability and practicality of the navigation map.

[0151] Through the embodiments of the present application, the actual image features detected by the robot during positioning are obtained, and the map image features corresponding to the positioning in the navigation map of the robot are obtained; when the matching degree between the actual image features and the map image features is less than the first threshold, multiple images of the positioning and multiple poses of the robot at the positioning are obtained; when the actual image features do not match the map image features due to changes in the actual environment, multiple images of the positioning related to the update of the navigation map and multiple poses of the robot at the positioning can be obtained to determine the timing of updating the navigation map, ensuring that the navigation map can be updated in a timely manner and improving the accuracy of navigation. Based on the multiple images and the multiple poses, correlation processing is performed to obtain a key-frame segment; the key-frame segment obtained by the correlation processing can accurately capture and reflect the continuous dynamic change information of the actual environment, and can provide more comprehensive and real-time environmental data support for the navigation and positioning of the robot. Based on the key-frame segment, the navigation map is updated to obtain an updated navigation map. The navigation map can be updated using the key-frame segment to maintain the timeliness and accuracy of the navigation map. Through the embodiments of the present application, the navigation map can be dynamically updated using the change information of the actual image features, ensuring the timeliness and accuracy of the navigation map, and enabling the robot to run stably for a long time.

[0152] Next, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0153] When a robot operates in a complex environment such as a factory, it needs to collect the change information of the environment according to its own operation situation, synchronously update the change information in the scene to the map, and at the same time delete the redundant environmental information to ensure the usability of the map and enable the robot to run stably for a long time. The embodiments of the present application are applicable to humanoid robots based on pure vision positioning and navigation.

[0154] In a complex factory environment, when a robot operates based on a map, due to environmental changes, such as: changes in lighting caused by different time points of the day, different seasons, and different weather conditions including sunny, rainy, and cloudy days; changing scenarios caused by certain structural changes in the environment, such as the change of objects in the environment during handling, there may be certain problems with positioning based on the original map, such as low positioning credibility or inability to position within a short period of time. In such cases, the environmental change information of the factory environment can be collected through the real-time positioning effect of the robot, and through a certain screening method, effective environmental change information can be obtained, and the effective environmental change information can be synchronously updated to the map, and useless information in the map can be synchronously deleted, so that the robot can operate and work stably for a long time.

[0155] Through the embodiments of the present application, the visual positioning ability of the humanoid robot in various factory scenarios can be improved, and the positioning requirements in environments with changing weather, lighting structure changes, etc. can be met. During operation, collecting the latest environmental change information and updating the map can improve the visual positioning ability and meet the long-term stable operation of the robot.

[0156] See Figure 4 , the robot operates in the established environmental map information 201, updates the positioning information in the environmental map information 201 according to the robot's running positioning situation 202. At the same time, according to the robot's running positioning situation 202, the environmental information acquisition module 203 collects the changing environmental information, conducts a certain screening, and then transmits the collected changing environmental information to the map update module 204. Finally, the map update module processes the new environmental information and updates the processed new environmental information to the environmental map information 201. And in order to ensure that the map file is not too large, the map update module needs to manage the map information and delete the old map information at the corresponding position in the map where the new environmental information is updated.

[0157] The environmental information acquisition module is mainly used to obtain the environmental change information in the map caused by reasons such as changes in lighting or structure for subsequent updating of the environmental map. See Figure 5, when the robot operates in the map, it collects visual information based on the positioning state of the robot. During the operation of the robot, when the matching degree between the image features of the robot positioning and the image features of the corresponding positioning in the map is less than a certain threshold, the four-eye image information and the pose information of the positioning of the robot are obtained. The pose information includes the position and the attitude, and the attitude mainly includes some angular information. The image features include the feature points extracted from the image, and the feature points mainly include corner points and descriptors. The corner points are the points that present obvious "corner" - shaped features. In the two - dimensional image space, the corner points are usually defined as the points where the image gray - value changes significantly in any direction. The pose information includes the position and the direction of the robot. The semantic dynamic objects are obtained according to the real - time object detection algorithm, and the images are screened. If the area of the semantic dynamic objects in the image is greater than a certain threshold, the image information is discarded, and only the images with fewer dynamic objects are retained. This is because the large proportion of dynamic objects in the image will affect the positioning accuracy and lead to positioning problems. The semantic dynamic object refers to an object that has dynamic features such as movement and change and is given a specific semantic meaning in a specific context. Here, "dynamic" not only refers to the physical movement of the object, but also includes the meaning change of the object in the context or the dynamic nature of people's cognition of it. The semantic dynamic objects are obtained through the real - time object detection algorithm; by comparing the changed image information and pose information collected last time, when it exceeds the threshold, the current image information and pose information are retained, and when it is less than the threshold, the image information and pose information collected this time are discarded to prevent excessive information collection. Finally, all the collected image information and pose information are transmitted to the map update module as environmental change information.

[0158] The map update module mainly processes the environmental change information obtained by the environmental information acquisition module, updates the processed environmental change information to the map, and deletes the old and useless information in the map according to the pose information. On the basis of maintaining the usability of the map, the size of the map is kept stable.

[0159] See Figure 6, the map update module 301 includes processing changed map information 302, redundancy processing 303, optimizing map information 304, and saving map information 305. For the processing of changed map information 302, the environmental change information is saved in the form of key frames. Feature points in the image are extracted, and triangulation is performed on the feature points in the image to obtain the positions and descriptors of the spatial feature points, which are applied to the positioning of the robot. Since the image information in the environmental change information changes continuously, it is necessary to process the continuously changing image information. The feature point information and pose information extracted from each image information constitute a key frame. The key frames that are temporally continuous are associated through feature point matching, that is, multiple spatial feature points can be observed by several key frames. The obtained key frame segment is added to the map. The addition method is based on the first key frame in the key frame segment to find the key frame in the map that is closest to the first key frame. The key frame segment can be added to the map through the connection with the key frame that is closest to the first key frame.. That is, the key frame segment is added behind the key frame with the closest pose. Among them, the map is a topological map, composed of the form of points and lines. The key frames in the key frame segment can be abstracted as the points of the topological map, and the connection relationship of the key frames can be abstracted as the edges of the topological map. The connection relationship of the key frames can be obtained by associating the temporally continuous key frames through feature point matching.

[0160] For redundancy processing 303, the map points in the old map near the added key frame segment, that is, the key frames in the old map with similar poses to the key frame segment, are counted. If it is impossible to perform positioning for a long time recently, that is, during the previous operation of the robot, it is statistically determined whether the image information and pose information collected by the current robot are consistent with the records in the map. If the gap is too large, it is considered that positioning is impossible, and if the number of key frames in the nearby old map is greater than the threshold, it indicates that there is redundancy in the key frames in the old map. Therefore, the key frames in the old map need to be deleted.

[0161] Optimize the map information 304, optimize the newly added key frame segments, perform map optimization through loop closure detection and feature matching. Use the key frames in the old map as observation information with unchanged poses, and optimize the pose information of the newly added key frame segments and the pose information of map points. Compare and delete map points with large errors. Loop closure detection is to align the old map and the new map by comparing the similarity of the key frames in the newly added key frame segments in the new map and the key frames in the old map in terms of image information, and use the key frame in the old map with the highest similarity and greater than a certain threshold as the loop closure detection key frame. Feature matching matches the feature points in the image information of the key frames corresponding to the loop closure detection key frame in the newly added key frame segments in the new map and the image information of the loop closure detection key frame, and infers the positions of the feature points in space based on the matching relationship. The main process of optimization is to optimize the pose information of the key frame segments and the pose information of map points. When performing map optimization, use the key frames in the old map as observation information with unchanged poses, take the positions of the feature points in space and the aligned map as inputs, and optimize the poses of the newly added key frames and the poses of map points, and compare and delete map points with large errors in the new map. If it is found that the pose information of some map points is inaccurate, i.e., the error is large, then delete the map points with large errors. Optimization is performed because when adding key frame segments to the map, the first key frame in the key frame segments is used to match the key frames in the old map. Therefore, the other key frames in the key frame segments may not be similar to the old map. When they are not similar, that is, the environment has changed, and the map needs to be updated.

[0162] Save the map information 305. After completing the update of the map, after the robot finishes its work, save the map information to a file. When applying later, the new map can be directly loaded, enabling the robot to run stably for a long time.

[0163] The embodiments of the present application can enable a humanoid robot to continuously update and iterate environmental information in the application of visual relocalization function, so as to achieve long-term positioning in various environments, with stability and robustness. In the application scenarios of humanoid robots, especially in scenarios such as factories, home services, and commercial services, actively obtain the latest and effective visual information through the operation process of the robot, thus solving the problem of visual positioning loss caused by changes in light and structure. Enable the humanoid robot to run for a long time and be applicable to more complex scenarios.

[0164] Next, continue to describe the exemplary structure of the software module implementation of the robot data processing device 455 provided by the embodiments of the present application. In some embodiments, refer to Figure 2 The software modules stored in the robot data processing device 455 in the memory 450 may include:

[0165] An acquisition module 4551, configured to acquire the actual image features detected during the positioning of the robot, and acquire the map image features corresponding to the positioning in the navigation map of the robot; when the matching degree between the actual image features and the map image features is less than a first threshold, acquire a plurality of images at the positioning and a plurality of poses of the robot at the positioning.

[0166] An association module 4552, configured to perform association processing based on the plurality of images and the plurality of poses to obtain a key frame segment.

[0167] An update module 4553, configured to update the navigation map based on the key frame segment to obtain an updated navigation map.

[0168] In some embodiments, the association module 4552 is further configured to screen out a plurality of static valid images from the plurality of images, and obtain valid poses corresponding to each of the static valid images from the plurality of poses; for each of the static valid images, extract feature points of the static valid image, and form a key frame corresponding to the static valid image by combining the feature points of the static valid image with the valid pose corresponding to the static valid image; perform association processing based on the key frames corresponding to each of the static valid images to obtain the key frame segment.

[0169] In some embodiments, the association module 4552 is further configured to group a plurality of the key frames to obtain at least one key frame combination, where a plurality of key frames included in the key frame combination have the same plurality of feature points; for each of the key frame combinations, associate the key frames included in the key frame combination in chronological order to obtain a key frame segment corresponding to the key frame combination.

[0170] In some embodiments, the association module 4552 is further configured to perform target recognition processing on each of the images to obtain the occupied area of the semantic dynamic object in each of the images; obtain a plurality of images with the occupied area less than a second threshold from the plurality of images as static images; for each of the static images, obtain a first difference between the static image and a historical static image corresponding to the static image; use the static image with the first difference greater than a third threshold as the static valid image, where the acquisition time of the historical static image is earlier than the acquisition time of the static image.

[0171] In some embodiments, the association module 4552 is further configured to obtain a second difference between the pose corresponding to the static image and the pose corresponding to the historical static image; when the second difference is less than a fourth threshold, compare the static image and the historical static image to obtain the first difference.

[0172] In some embodiments, the update module 4553 is further configured to add the key-frame segment to the navigation map to obtain a first intermediate navigation map; delete redundant map points corresponding to the key-frame segment from the first intermediate navigation map to obtain a second intermediate navigation map; and optimize the key-frame segment in the second intermediate navigation map based on the navigation map to obtain an updated navigation map.

[0173] In some embodiments, the update module 4553 is further configured to match the pose corresponding to the first key frame in the key-frame segment with the poses corresponding to multiple map points in the navigation map to obtain multiple first matching values; use the map point corresponding to the largest first matching value among the multiple first matching values as the first matching map point; generate new map points corresponding to each key frame in the key-frame segment, and generate new map edges based on the association relationship between the key frames in the key-frame segment; and add the new map points and the new map edges to the first matching map point in the navigation map to obtain the first intermediate navigation map.

[0174] In some embodiments, the update module 4553 is further configured to obtain a third difference between the pose corresponding to each map point in the first intermediate navigation map and the pose corresponding to the key-frame segment; use the map points corresponding to third differences less than a fifth threshold as redundant map points to be processed; obtain a fourth difference between the redundant map points to be processed and historical key frames, where the historical key frames are obtained by the robot at the positions corresponding to the redundant map points to be processed; use the redundant map points to be processed corresponding to fourth differences greater than a sixth threshold as the redundant map points; and the step of deleting the redundant map points corresponding to the key-frame segment from the first intermediate navigation map to obtain a second intermediate navigation map includes deleting the redundant map points in the first intermediate navigation map when the number of the redundant map points is greater than a seventh threshold to obtain the second intermediate navigation map.

[0175] In some embodiments, the update module 4553 is further configured to obtain a second matching map point that matches the key-frame segment from multiple map points in the navigation map; align the navigation map and the second intermediate navigation map based on the second matching map point and the key frame in the key-frame segment corresponding to the second matching map point to obtain a third intermediate navigation map; perform feature matching processing on the image features corresponding to the second matching map point and the image features of the key frame in the key-frame segment corresponding to the second matching map point to obtain a feature matching result; and optimize the key-frame segment in the second intermediate navigation map based on the feature matching result to obtain the updated navigation map.

[0176] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium, and the processor executes the computer-executable instructions, so that the electronic device executes the data processing method of the robot in the above embodiment of the present application.

[0177] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions or a computer program are stored. When the computer-executable instructions or the computer program are executed by a processor, the processor will be caused to execute the data processing method of the robot provided in the embodiment of the present application.

[0178] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0179] In some embodiments, the computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0180] As an example, the computer-executable instructions may or may not correspond to files in the file system, and may be stored as part of a file that stores other programs or data. For example, they may be stored in one or more scripts in a Hyper Text Markup Language (HTML) document, stored in a single file dedicated to the program being discussed, or stored in multiple cooperating files (for example, files that store one or more modules, subroutines, or code portions).

[0181] As an example, the computer-executable instructions may be deployed to execute on one electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed at multiple locations and interconnected by a communication network.

[0182] In summary, through the embodiments of the present application, when the actual image features do not match the map image features due to changes in the actual environment, multiple images related to the positioning for updating the navigation map and multiple poses of the robot during positioning can be obtained, the timing for updating the navigation map can be determined, ensuring that the navigation map can be updated in a timely manner and improving the accuracy of navigation. The key frame segments obtained through correlation processing can accurately capture and reflect the continuous dynamic change information of the actual environment, providing more comprehensive and real-time environmental data support for the navigation and positioning of the robot. Images with similar poses but large content differences can be determined, improving the image comparison efficiency. The subtle differences between images can be accurately captured, improving the matching accuracy. Only when the differences between poses meet the conditions, the differences between images are compared, which can reduce unnecessary image comparison operations and save computing resources and time costs. Images with an occupancy area smaller than the second threshold are selected as static images, effectively reducing the interference of semantic dynamic objects on the update of the navigation map. Static images with significant scene changes can be accurately identified and extracted, while reducing the computational complexity and improving the processing efficiency. By screening out static valid images, the obtained key frame segments can efficiently integrate images and poses, accurately capture the dynamic changes in the scene, and improve the pertinence and efficiency of image processing. Grouping multiple key frames, combining the key frames together based on the same feature points, and associating these key frames based on the time sequence to form key frame segments can efficiently integrate and utilize the information in the images. It can not only accurately capture the static and dynamic changes in the scene, but also improve the efficiency and accuracy of image processing, significantly enhancing the practicability and robustness of the system. Matching the pose of the first frame of the key frame segment with the map point pose in the navigation map, selecting the optimal matching point as the insertion position of the key frame segment, generating new map points and map edges from each key frame in the key frame segment and the association relationship between the key frames, and then integrating them into the navigation map can dynamically update the navigation map and ensure the timeliness and accuracy of the map information. It not only enhances the integrity and coherence of the map, but also provides a more reliable environmental model for navigation, significantly improving the positioning accuracy and path planning ability of the navigation map. The navigation map can be updated using the key frame segments to maintain the real-time nature and accuracy of the navigation map. Redundant information in the navigation map can be accurately removed, avoiding the expansion of the navigation map and positioning errors. When the number of redundant map points exceeds the set threshold, the redundant map points are automatically deleted to ensure the simplicity and efficiency of the navigation map, providing a more accurate and reliable navigation environment for the robot, and significantly improving the stability and practicability of the navigation map. After adding the key frame segments to the navigation map and removing the redundant points, optimizing the key frame segments can significantly improve the accuracy and practicability of the navigation map. The optimized navigation map provides clearer and more accurate navigation guidance for the robot, significantly improving the efficiency and safety of navigation.Through the embodiments of the present application, it is possible to dynamically update the navigation map using the change information of the actual image features, ensuring the timeliness and accuracy of the navigation map, so that the robot can operate stably for a long time.

[0183] As described above, the above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and scope of the present application are all included in the protection scope of the present application.

Claims

1. A data processing method for a robot, characterized in that The method includes: Obtaining the actual image features detected by the robot during positioning, and obtaining the map image features corresponding to the positioning in the navigation map of the robot; When the matching degree between the actual image features and the map image features is less than a first threshold, obtaining a plurality of images of the positioning and a plurality of poses of the robot at the positioning; Performing association processing based on the plurality of images and the plurality of poses to obtain a key frame segment; Updating the navigation map based on the key frame segment to obtain an updated navigation map.

2. The method according to claim 1, characterized in that, The performing association processing based on the plurality of images and the plurality of poses to obtain a key frame segment includes: Filtering out a plurality of static valid images from the plurality of images, and obtaining valid poses corresponding to each of the static valid images from the plurality of poses; For each of the static valid images, extracting feature points of the static valid image, and forming a key frame corresponding to the static valid image by combining the feature points of the static valid image with the valid pose corresponding to the static valid image; Performing association processing based on the key frames corresponding to each of the static valid images to obtain the key frame segment.

3. The method according to claim 2, characterized in that, The performing association processing based on the key frames corresponding to each of the static valid images to obtain the key frame segment includes: Grouping a plurality of the key frames to obtain at least one key frame combination, where a plurality of key frames included in the key frame combination have the same plurality of feature points; For each of the key frame combinations, associating the key frames included in the key frame combination in chronological order to obtain a key frame segment corresponding to the key frame combination.

4. The method according to claim 2, characterized in that The filtering out a plurality of static valid images from the plurality of images includes: Performing target recognition processing on each of the images to obtain the occupancy area of the semantic dynamic object in each of the images; Obtaining a plurality of images with an occupancy area less than a second threshold from the plurality of images as static images; For each of the static images, obtaining a first difference between the static image and a historical static image corresponding to the static image; Taking the static image with the first difference greater than a third threshold as the static valid image, where the acquisition time of the historical static image is earlier than the acquisition time of the static image.

5. The method according to claim 4, characterized in that, The obtaining a first difference between the static image and a historical static image corresponding to the static image includes: Obtaining a second difference between the pose corresponding to the static image and the pose corresponding to the historical static image; When the second difference is less than a fourth threshold, comparing the static image and the historical static image to obtain the first difference.

6. The method according to claim 1, wherein The updating the navigation map based on the key frame segment to obtain an updated navigation map includes: Adding the key frame segment to the navigation map to obtain a first intermediate navigation map; Deleting redundant map points corresponding to the key frame segment from the first intermediate navigation map to obtain a second intermediate navigation map; Optimizing the key frame segment in the second intermediate navigation map based on the navigation map to obtain an updated navigation map.

7. The method according to claim 6, wherein The key-frame segment includes a plurality of key frames. Adding the key-frame segment to the navigation map to obtain a first intermediate navigation map includes: Matching the pose corresponding to the first key frame in the key-frame segment with the poses corresponding to a plurality of map points in the navigation map to obtain a plurality of first matching values; Taking the map point corresponding to the largest first matching value among the plurality of first matching values as the first matching map point; Generating new map points corresponding to each key frame in the key-frame segment, and generating new map edges based on the association relationship between the key frames in the key-frame segment; Adding the new map points and the new map edges to the first matching map point in the navigation map to obtain the first intermediate navigation map.

8. The method according to claim 6, wherein The method further includes: Obtaining a third difference between the pose corresponding to each map point in the first intermediate navigation map and the pose corresponding to the key-frame segment; Taking the map points corresponding to the third differences less than a fifth threshold as redundant map points to be processed; Obtaining a fourth difference between the redundant map points to be processed and historical key frames, where the historical key frames are obtained by the robot at the positioning corresponding to the redundant map points to be processed; Taking the redundant map points to be processed corresponding to the fourth differences greater than a sixth threshold as the redundant map points; Deleting the redundant map points corresponding to the key-frame segment from the first intermediate navigation map to obtain a second intermediate navigation map, including When the number of the redundant map points is greater than a seventh threshold, deleting the redundant map points in the first intermediate navigation map to obtain the second intermediate navigation map.

9. The method according to claim 6, wherein Optimizing the key-frame segment in the second intermediate navigation map based on the navigation map to obtain an updated navigation map, including: Obtaining a second matching map point that matches the key-frame segment from the plurality of map points in the navigation map; Aligning the navigation map and the second intermediate navigation map based on the second matching map point and the key frame in the key-frame segment corresponding to the second matching map point to obtain a third intermediate navigation map; Performing feature matching processing on the image features corresponding to the second matching map point and the image features of the key frame in the key-frame segment corresponding to the second matching map point to obtain a feature matching result; Optimizing the key-frame segment in the second intermediate navigation map based on the feature matching result to obtain the updated navigation map.

10. A data processing device for a robot, characterized in that, The apparatus includes: An acquisition module, configured to acquire the actual image features detected by the robot during positioning, and acquire the map image features corresponding to the positioning in the navigation map of the robot; when the matching degree between the actual image features and the map image features is less than a first threshold, acquire a plurality of images of the positioning and a plurality of poses of the robot at the positioning; An association module, configured to perform association processing based on the plurality of images and the plurality of poses to obtain a key-frame segment; An update module, configured to update the navigation map based on the key-frame segment to obtain an updated navigation map.

11. An electronic device, characterized in that, The electronic device includes: a memory for storing computer-executable instructions or a computer program; a processor for implementing the method according to any one of claims 1 to 9 when executing the computer-executable instructions or the computer program stored in the memory.

12. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, The computer-executable instructions or the computer program, when executed by the processor, implement the method according to any one of claims 1 to 9.

Images