Visual repositioning method and device, computer readable storage medium and robot

Abstract

The invention belongs to the technical field of robots, and particularly relates to a visual repositioning method and device, a computer readable storage medium and a robot. The method comprises the steps of performing semantic perception on a first image collected by a visual sensor of a robot under the condition that visual relocation fails to obtain visual semantic perception information; determining angle adjustment information of the visual sensor according to the visual semantic perception information; performing angle adjustment on the visual sensor according to the angle adjustment information; and performing visual repositioning on the robot according to a second image acquired by the visual sensor after angle adjustment to obtain a visual repositioning result. According to the visual repositioning method and device, the angle adjustment information of the visual sensor can be determined based on the visual semantic perception information, rich information in a wider visual field is obtained through angle adjustment, visual repositioning is carried out on the basis, the phenomenon of repositioning loss can be effectively reduced, and the stability of visual repositioning is improved.

Description

Technical Field

[0001] The present application belongs to the field of robotics technology, and in particular, relates to a visual relocalization method, device, computer-readable storage medium, and robot. Background Art

[0002] In the field of robotics, visual relocalization is a very important basic problem. Visual relocalization refers to the problem of determining the robot's position and posture in an existing visual prior map based on the data captured in real time by the visual sensor. In the prior art, there are many visual relocalization methods, but these existing visual relocalization methods are often limited by the limited information under a fixed field of view, are prone to relocalization loss, and have poor stability. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a visual repositioning method, device, computer-readable storage medium and robot to solve the problems of easy repositioning loss and poor stability in existing visual repositioning methods.

[0004] A first aspect of an embodiment of the present application provides a visual repositioning method, which may include:

[0005] In the case where visual relocalization fails, semantic perception is performed on the first image captured by the robot's visual sensor to obtain visual semantic perception information;

[0006] Determining angle adjustment information of the visual sensor according to the visual semantic perception information;

[0007] Adjusting the angle of the visual sensor according to the angle adjustment information;

[0008] According to the second image collected by the visual sensor after the angle adjustment, the robot is visually relocated to obtain a visual relocation result.

[0009] In a specific implementation of the first aspect, determining the angle adjustment information of the visual sensor according to the visual semantic perception information may include:

[0010] Determining a scene difference between the visual semantic perception information and preset visual prior map information;

[0011] If the scene difference is greater than a preset scene difference threshold, determining a first angle limit and a second angle limit of the visual sensor; wherein the first angle limit is an angle limit in a first direction, the second angle limit is an angle limit in a second direction, and the first direction and the second direction are opposite directions;

[0012] An angle traversal from the first angle limit to the second angle limit is determined as the angle adjustment information.

[0013] In a specific implementation of the first aspect, after determining the scene difference between the visual semantic perception information and the preset visual priori map information, the method may further include:

[0014] If the scene difference is less than or equal to the scene difference threshold, determining the bias angle of the visual sensor according to the visual semantic perception information;

[0015] The deflection angle is determined as the angle adjustment information.

[0016] In a specific implementation of the first aspect, determining the bias angle of the visual sensor according to the visual semantic perception information may include:

[0017] Determining the regional scene differences of various regions in the field of vision of the robot respectively according to the visual semantic perception information;

[0018] Selecting a first target area according to the regional scene difference, wherein the first target area is the area with the smallest regional scene difference;

[0019] The angle corresponding to the first target area is determined as the deflection angle.

[0020] In a specific implementation of the first aspect, determining the bias angle of the visual sensor according to the visual semantic perception information may include:

[0021] Determining a static semantic object in the field of view of the robot according to the visual semantic perception information;

[0022] The angle corresponding to the static semantic object is determined as the bias angle.

[0023] In a specific implementation of the first aspect, determining the bias angle of the visual sensor according to the visual semantic perception information may include:

[0024] Determining the texture richness of each area in the robot's field of view according to the visual semantic perception information;

[0025] Selecting a second target area according to the texture richness, wherein the second target area is an area with the greatest texture richness;

[0026] The angle corresponding to the second target area is determined as the deflection angle.

[0027] In a specific implementation of the first aspect, after visually relocating the robot to obtain a visual relocation result, the method may further include:

[0028] If the visual relocalization result is a visual relocalization failure, the process returns to the step of performing semantic perception on the first image acquired by the visual sensor of the robot and subsequent steps thereof until the visual relocalization result is a visual relocalization success.

[0029] A second aspect of the embodiments of the present application provides a visual repositioning device, which may include:

[0030] A visual semantic perception module, used to perform semantic perception on the first image collected by the robot's visual sensor to obtain visual semantic perception information when visual relocalization fails;

[0031] An angle adjustment information determination module, used to determine the angle adjustment information of the visual sensor according to the visual semantic perception information;

[0032] An angle adjustment module, used for adjusting the angle of the visual sensor according to the angle adjustment information;

[0033] The visual repositioning module is used to perform visual repositioning on the robot according to the second image collected by the visual sensor after the angle adjustment to obtain a visual repositioning result.

[0034] In a specific implementation of the second aspect, the angle adjustment information determination module may include:

[0035] A scene difference determination submodule, used to determine the scene difference between the visual semantic perception information and the preset visual prior map information;

[0036] An angle traversal adjustment information determination submodule is used to determine the first angle limit and the second angle limit of the visual sensor if the scene difference is greater than a preset scene difference threshold; wherein the first angle limit is the angle limit in a first direction, the second angle limit is the angle limit in a second direction, and the first direction and the second direction are opposite directions; and the angle traversal from the first angle limit to the second angle limit is determined as the angle adjustment information.

[0037] In a specific implementation of the second aspect, the angle adjustment information determination module may further include:

[0038] The deflection angle adjustment information determination submodule is used to determine the deflection angle of the visual sensor according to the visual semantic perception information if the scene difference is less than or equal to the scene difference threshold; and determine the deflection angle as the angle adjustment information.

[0039] In a specific implementation of the second aspect, the deflection angle adjustment information determination submodule may include:

[0040] The first bias angle determination unit is used to determine the regional scene differences of each area in the robot's field of view according to the visual semantic perception information; select a first target area according to the regional scene differences, wherein the first target area is the area with the smallest regional scene difference; and determine the angle corresponding to the first target area as the bias angle.

[0041] In a specific implementation of the second aspect, the deflection angle adjustment information determination submodule may include:

[0042] The second bias angle determination unit is used to determine the static semantic object in the field of view of the robot according to the visual semantic perception information; and determine the angle corresponding to the static semantic object as the bias angle.

[0043] In a specific implementation of the second aspect, the deflection angle adjustment information determination submodule may include:

[0044] The third bias angle determination unit is used to determine the texture richness of each area in the robot's field of view according to the visual semantic perception information; select a second target area according to the texture richness, wherein the second target area is the area with the largest texture richness; and determine the angle corresponding to the second target area as the bias angle.

[0045] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned visual repositioning methods are implemented.

[0046] A fourth aspect of an embodiment of the present application provides a robot, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned visual repositioning methods when executing the computer program.

[0047] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product runs on a robot, the robot executes the steps of any one of the above-mentioned visual relocalization methods.

[0048] Compared with the prior art, the embodiments of the present application have the following beneficial effects: when visual relocation fails, the embodiments of the present application perform semantic perception on the first image captured by the robot's visual sensor to obtain visual semantic perception information; determine the angle adjustment information of the visual sensor based on the visual semantic perception information; adjust the angle of the visual sensor based on the angle adjustment information; perform visual relocation on the robot based on the second image captured by the visual sensor after the angle adjustment to obtain a visual relocation result. Through the embodiments of the present application, the angle adjustment information of the visual sensor can be determined based on the visual semantic perception information, and rich information in a wider field of view can be obtained through angle adjustment. Visual relocation is performed on this basis, which can effectively reduce the phenomenon of relocation loss and improve the stability of visual relocation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is a flow chart of an embodiment of a visual repositioning method in an embodiment of the present application;

[0051] Figure 2 A schematic flow chart for determining angle adjustment information of a visual sensor based on visual semantic perception information;

[0052] Figure 3 is a schematic diagram of the rotational degrees of freedom of the robot's head;

[0053] Figure 4 A schematic diagram for angle traversal;

[0054] Figure 5 This is a structural diagram of an embodiment of a visual repositioning device in an embodiment of the present application;

[0055] Figure 6 This is a schematic block diagram of a robot in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0058] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0059] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0060] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0061] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0062] In the field of robotics, visual relocalization is a very important basic problem. Visual relocalization refers to the problem of determining the robot's position and posture in an existing visual prior map based on the data captured in real time by the visual sensor. In the prior art, there are many visual relocalization methods, but these existing visual relocalization methods are often limited by the limited information under a fixed field of view, are prone to relocalization loss, and have poor stability.

[0063] In view of this, the embodiments of the present application provide a visual repositioning method, device, computer-readable storage medium and robot to solve the problems of easy repositioning loss and poor stability in existing visual repositioning methods.

[0064] In an embodiment of the present application, the angle adjustment information of the visual sensor can be determined based on the visual semantic perception information, and rich information under a wider field of view can be obtained through angle adjustment. Visual repositioning is performed on this basis, which can effectively reduce the phenomenon of repositioning loss and improve the stability of visual repositioning.

[0065] The execution subject of the embodiments of the present application may be a robot, including but not limited to industrial robots, household service robots, commercial service robots and other various types of robots.

[0066] See also Figure 1 , an embodiment of a visual repositioning method in an embodiment of the present application may include:

[0067] Step S101: When visual relocalization fails, semantic perception is performed on a first image captured by a visual sensor of the robot to obtain visual semantic perception information.

[0068] The visual relocalization function can usually be implemented by the robot's Visual Simultaneous Localization and Mapping (VSLAM) module, which can include states such as first relocalization, continuous frame positioning, and relocalization. Each state can include both success and failure.

[0069] When the robot starts to perform a task, it first performs the first relocalization, matching the image collected by the robot's visual sensor with the preset visual prior map, so as to obtain the robot's position and posture in the map coordinate system and realize the relocalization function.

[0070] While the robot is performing a task, it can perform continuous frame positioning. At this time, the robot's visual sensor will continue to collect images and match these images with the visual prior map in real time to obtain real-time relocalization results.

[0071] In the case of continuous frame positioning loss, the robot can obtain the short-term posture by using devices such as inertial measurement unit (IMU) and kinematic odometer, but there are long-term cumulative errors. Therefore, the robot can perform re-positioning, and the process of re-positioning is the same as the first repositioning.

[0072] If the robot can successfully perform visual relocalization through the above process, the visual sensor angle can be adjusted without any adjustment, and the subsequent visual relocalization can still be performed according to the above process. If the robot's visual relocalization fails, the visual sensor angle can be adjusted to obtain rich information in a wider field of view, and visual relocalization can be performed on this basis.

[0073] In order to make more targeted angle adjustments to the visual sensor, the embodiment of the present application can perform semantic perception on the image (referred to as the first image) collected by the robot's visual sensor when visual relocalization fails, thereby obtaining corresponding visual semantic perception information. The visual semantic perception information may include but is not limited to semantic objects, text information, and texture information.

[0074] Taking semantic objects as an example, the first image can be semantically segmented by a preset semantic segmentation model to obtain various semantic objects in the first image, including but not limited to pedestrians, vehicles, tables, cabinets and other types of semantic objects. The specific semantic segmentation model to be used can be flexibly set according to actual conditions, and may include but not limited to YOLO, U-Net, FCN, SegNet, PSPNet, DeepLab and other models, which are not specifically limited in the embodiments of the present application.

[0075] Taking text information as an example, the first image can be subjected to text recognition based on a preset optical character recognition (OCR) algorithm, thereby obtaining text information in the first image, including but not limited to various types of text information such as Chinese and English. The specific optical character recognition algorithm to be used can be flexibly set according to actual conditions, and may include but not limited to a rule-based recognition algorithm, a template matching-based recognition algorithm, a deep learning-based recognition algorithm, and other recognition algorithms, which are not specifically limited in the embodiments of the present application.

[0076] Taking texture information as an example, feature point detection can be performed on the first image based on a preset feature point detection algorithm to obtain feature points with significant features in the first image, such as corner points, edge points or other significant texture points. The specific feature point detection algorithm used can be flexibly set according to actual conditions, and may include but is not limited to Harris, SIFT, SURF, ORB, FAST, BRISK and other detection algorithms, which are not specifically limited in the embodiments of the present application.

[0077] Step S102: determining angle adjustment information of the visual sensor according to the visual semantic perception information.

[0078] like Figure 2As shown, step S102 may include the following process:

[0079] Step S1021: Determine the scene difference between the visual semantic perception information and the preset visual priori map information.

[0080] In an embodiment of the present application, feature point matching can be performed between the feature points in the visual semantic perception information and the feature points in the visual prior map information, and the ratio between the number of feature points that failed to match in the visual semantic perception information and the total number of feature points is determined as the scene difference.

[0081] Step S1022: determine whether the scene difference is greater than a preset scene difference threshold.

[0082] The specific value of the scene difference threshold can be flexibly set according to the actual situation, and the embodiments of the present application do not make specific limitations on this. If the scene difference is greater than the preset scene difference threshold, it means that the scene difference is too large, and the current visual semantic perception information has little reference significance. At this time, an unbiased angle traversal adjustment can be performed, that is, step S1023 is executed. If the scene difference is less than or equal to the scene difference threshold, it means that the scene difference is not large, and the current visual semantic perception information still has reference significance. At this time, a biased angle adjustment can be performed based on the visual semantic perception information, that is, step S1024 is executed.

[0083] Step S1023: determine a first angle limit and a second angle limit of the visual sensor, and determine the angle traversal from the first angle limit to the second angle limit as angle adjustment information.

[0084] The first angle limit is the angle limit in the first direction, the second angle limit is the angle limit in the second direction, and the first direction and the second direction are opposite directions.

[0085] When the vision sensor is located on the head of the robot, the vision sensor will move with the head, such as Figure 3 As shown, the robot's head can include three rotational degrees of freedom, namely the pitch angle around the Y axis, the yaw angle around the Z axis, and the roll angle around the X axis, wherein the front direction of the head is the positive direction of the X axis, the right direction of the head is the positive direction of the Y axis, and the upward direction of the head is the positive direction of the Z axis. The visual sensor can rotate with the rotation of the robot's head in the three rotational degrees of freedom.

[0086] In the embodiment of the present application, the field of view of the robot can be changed by adjusting the yaw angle to perform visual repositioning under the new field of view. The first direction can be a direction of rotation to the left, and the second direction can be a direction of rotation to the right. The first angle limit is the angle limit for rotation to the left, and the angle value is a negative value. The second angle limit is the angle limit for rotation to the right, and the angle value is a positive value. The specific values ​​of the first angle limit and the second angle limit can be flexibly set according to actual conditions, and the embodiment of the present application does not make specific restrictions on this. As an example, the first angle limit can be set to -90 degrees, and the second angle limit can be set to 90 degrees. In this case, if Figure 4 As shown, the head of the robot can be rotated from the leftmost to the rightmost step by step, that is, the angle traversal between -90 degrees and 90 degrees is determined as the angle adjustment information. Of course, the opposite setting can also be performed, that is, the head of the robot can be rotated from the rightmost to the leftmost step by step, that is, the angle traversal between 90 degrees and -90 degrees is determined as the angle adjustment information.

[0087] Step S1024: determine the bias angle of the visual sensor according to the visual semantic perception information, and determine the bias angle as the angle adjustment information.

[0088] In the embodiment of the present application, the bias angle of the visual sensor may be determined based on at least one of scene difference, static semantic objects, and texture richness.

[0089] Taking scene differences as an example, the regional scene differences of each area in the robot's field of view can be determined separately according to the visual semantic perception information. The specific division method of each area can be flexibly set according to the actual situation, and the embodiment of the present application does not specifically limit this. For example, the robot's field of view can be divided equally from left to right to divide each area. For any area, the ratio between the number of feature points that failed to match in the area and the total number of feature points in the area can be determined as the regional scene difference. After determining the regional scene differences of each area, the first target area can be selected according to the regional scene differences, and the angle corresponding to the first target area is determined as the bias angle. Among them, the first target area is the area with the smallest regional scene difference. If the robot adjusts the angle of the visual sensor according to this bias angle, the visual sensor can be directed to the area with the smallest scene difference, which can effectively increase the probability of successful relocation.

[0090] Taking static semantic objects as an example, the static semantic objects in the robot's field of view can be determined based on the visual semantic perception information, and the angle corresponding to the static semantic objects can be determined as the bias angle. Among them, static semantic objects can include but are not limited to fixed-position texts, tables, cabinets and other objects. If the robot adjusts the angle of the visual sensor according to this bias angle, the visual sensor can be directed toward the static semantic object with a fixed position, which can effectively increase the probability of successful relocation.

[0091] Taking texture richness as an example, the texture richness of each area in the robot's field of view can be determined based on the visual semantic perception information. For any area, the total number of feature points in the area can be determined as the texture richness of the area. After determining the texture richness of each area, the second target area can be selected based on the texture richness, and the angle corresponding to the second target area can be determined as the bias angle. Among them, the second target area is the area with the greatest texture richness. If the robot adjusts the angle of the visual sensor according to this bias angle, the visual sensor can be directed toward the area with the greatest texture richness, which can effectively increase the probability of successful relocation.

[0092] Scene differences, static semantic objects, and texture richness may have different priorities. For example, scene differences may be set to the highest priority, static semantic objects may be set to the second highest priority, and texture richness may be set to the lowest priority, that is, the bias angle is determined based on scene differences first, followed by the bias angle based on static semantic objects, and then the bias angle is determined based on texture richness. Of course, other different priority settings may also be made according to actual conditions, and the embodiments of the present application do not specifically limit this.

[0093] Step S103: adjusting the angle of the visual sensor according to the angle adjustment information.

[0094] In the case where the visual sensor is located at the head of the robot, if the angle adjustment information is angle traversal, the head of the robot can be controlled to rotate gradually from the leftmost to the rightmost, or from the rightmost to the leftmost, thereby driving the visual sensor to rotate gradually. If the angle adjustment information is a bias angle, the head of the robot can be controlled to rotate to the bias angle, thereby driving the visual sensor to rotate to the bias angle.

[0095] Step S104: Perform visual repositioning on the robot according to the second image captured by the visual sensor after the angle adjustment to obtain a visual repositioning result.

[0096] Since the angle of the robot's visual sensor is adjusted, the robot's field of view changes accordingly. The embodiment of the present application can re-position the robot visually based on the image (recorded as the second image) collected by the visual sensor in the new field of view, thereby obtaining the corresponding visual repositioning result. If the visual repositioning result is a visual repositioning failure, then the robot can be re-positioned according to the following method: Figure 1 The process shown readjusts the angle and re-performs visual relocation under the new field of view until the visual relocation result is successful.

[0097] To sum up, in the embodiments of the present application, the angle adjustment information of the visual sensor can be determined based on the visual semantic perception information, and rich information under a wider field of view can be obtained through angle adjustment. Visual repositioning is performed on this basis, which can effectively reduce the phenomenon of repositioning loss and improve the stability of visual repositioning.

[0098] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0099] Corresponding to a visual repositioning method described in the above embodiment, Figure 5 A structural diagram of an embodiment of a visual repositioning device provided in an embodiment of the present application is shown.

[0100] In this embodiment, a visual relocation device may include:

[0101] The visual semantic perception module 501 is used to perform semantic perception on the first image collected by the robot's visual sensor to obtain visual semantic perception information when visual relocalization fails;

[0102] An angle adjustment information determination module 502, configured to determine angle adjustment information of the visual sensor according to the visual semantic perception information;

[0103] An angle adjustment module 503, used to adjust the angle of the visual sensor according to the angle adjustment information;

[0104] The visual repositioning module 504 is used to perform visual repositioning on the robot according to the second image collected by the visual sensor after the angle adjustment to obtain a visual repositioning result.

[0105] In a specific implementation of the embodiment of the present application, the angle adjustment information determination module may include:

[0106] A scene difference determination submodule, used to determine the scene difference between the visual semantic perception information and the preset visual prior map information;

[0107] An angle traversal adjustment information determination submodule is used to determine the first angle limit and the second angle limit of the visual sensor if the scene difference is greater than a preset scene difference threshold; wherein the first angle limit is the angle limit in a first direction, the second angle limit is the angle limit in a second direction, and the first direction and the second direction are opposite directions; and the angle traversal from the first angle limit to the second angle limit is determined as the angle adjustment information.

[0108] In a specific implementation of the embodiment of the present application, the angle adjustment information determination module may further include:

[0109] The deflection angle adjustment information determination submodule is used to determine the deflection angle of the visual sensor according to the visual semantic perception information if the scene difference is less than or equal to the scene difference threshold; and determine the deflection angle as the angle adjustment information.

[0110] In a specific implementation of the embodiment of the present application, the deflection angle adjustment information determination submodule may include:

[0111] The first bias angle determination unit is used to determine the regional scene differences of each area in the robot's field of view according to the visual semantic perception information; select a first target area according to the regional scene differences, wherein the first target area is the area with the smallest regional scene difference; and determine the angle corresponding to the first target area as the bias angle.

[0112] In a specific implementation of the embodiment of the present application, the deflection angle adjustment information determination submodule may include:

[0113] The second bias angle determination unit is used to determine the static semantic object in the field of view of the robot according to the visual semantic perception information; and determine the angle corresponding to the static semantic object as the bias angle.

[0114] In a specific implementation of the embodiment of the present application, the deflection angle adjustment information determination submodule may include:

[0115] The third bias angle determination unit is used to determine the texture richness of each area in the robot's field of view according to the visual semantic perception information; select a second target area according to the texture richness, wherein the second target area is the area with the largest texture richness; and determine the angle corresponding to the second target area as the bias angle.

[0116] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0117] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Figure 6 A schematic block diagram of a robot provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0119] like Figure 6 As shown, the robot 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned various visual relocation method embodiments are implemented, for example Figure 1 Alternatively, when the processor 60 executes the computer program 62, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of modules 501 to 504 are shown.

[0120] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 62 in the robot 6.

[0121] Those skilled in the art will understand that Figure 6 It is only an example of the robot 6 and does not constitute a limitation of the robot 6. The robot 6 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the robot 6 may also include input and output devices, network access devices, buses, etc.

[0122] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0123] The memory 61 may be an internal storage unit of the robot 6, such as a hard disk or memory of the robot 6. The memory 61 may also be an external storage device of the robot 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the robot 6. Further, the memory 61 may include both an internal storage unit and an external storage device of the robot 6. The memory 61 is used to store the computer program and other programs and data required by the robot 6. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0124] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0125] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0126] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0127] In the embodiments provided in the present application, it should be understood that the disclosed devices / robots and methods can be implemented in other ways. For example, the device / robot embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0131] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A visual relocalization method, characterized in that: include: In the case where visual relocalization fails, semantic perception is performed on the first image captured by the robot's visual sensor to obtain visual semantic perception information; Determining angle adjustment information of the visual sensor according to the visual semantic perception information; Adjusting the angle of the visual sensor according to the angle adjustment information; According to the second image collected by the visual sensor after the angle adjustment, the robot is visually relocated to obtain a visual relocation result.

2. The visual relocation method according to claim 1, characterized in that: The determining the angle adjustment information of the visual sensor according to the visual semantic perception information includes: Determining a scene difference between the visual semantic perception information and preset visual prior map information; If the scene difference is greater than a preset scene difference threshold, determining a first angle limit and a second angle limit of the visual sensor; wherein the first angle limit is an angle limit in a first direction, the second angle limit is an angle limit in a second direction, and the first direction and the second direction are opposite directions; An angle traversal from the first angle limit to the second angle limit is determined as the angle adjustment information.

3. The visual relocation method according to claim 2, characterized in that: After determining the scene difference between the visual semantic perception information and the preset visual priori map information, the method further includes: If the scene difference is less than or equal to the scene difference threshold, determining the bias angle of the visual sensor according to the visual semantic perception information; The deflection angle is determined as the angle adjustment information.

4. The visual relocation method according to claim 3, characterized in that: The determining the bias angle of the visual sensor according to the visual semantic perception information comprises: Determining the regional scene differences of various regions in the field of vision of the robot respectively according to the visual semantic perception information; Selecting a first target area according to the regional scene difference, wherein the first target area is the area with the smallest regional scene difference; The angle corresponding to the first target area is determined as the deflection angle.

5. The visual relocation method according to claim 3, characterized in that: The determining the bias angle of the visual sensor according to the visual semantic perception information comprises: Determining a static semantic object in the field of view of the robot according to the visual semantic perception information; The angle corresponding to the static semantic object is determined as the bias angle.

6. The visual relocation method according to claim 3, characterized in that: The determining the bias angle of the visual sensor according to the visual semantic perception information comprises: Determining the texture richness of each area in the robot's field of view according to the visual semantic perception information; Selecting a second target area according to the texture richness, wherein the second target area is an area with the greatest texture richness; The angle corresponding to the second target area is determined as the deflection angle.

7. The visual relocation method according to any one of claims 1 to 6, characterized in that: After visual repositioning of the robot is performed to obtain a visual repositioning result, the method further includes: If the visual relocalization result is a visual relocalization failure, the process returns to the step of performing semantic perception on the first image acquired by the visual sensor of the robot and subsequent steps thereof until the visual relocalization result is a visual relocalization success.

8. A visual repositioning device, characterized in that: include: A visual semantic perception module, used to perform semantic perception on the first image acquired by the robot's visual sensor to obtain visual semantic perception information when visual relocalization fails; An angle adjustment information determination module, used to determine the angle adjustment information of the visual sensor according to the visual semantic perception information; An angle adjustment module, used for adjusting the angle of the visual sensor according to the angle adjustment information; The visual repositioning module is used to perform visual repositioning on the robot according to the second image collected by the visual sensor after the angle adjustment to obtain a visual repositioning result.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the visual relocalization method according to any one of claims 1 to 7 are implemented.

10. A robot comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the visual relocalization method according to any one of claims 1 to 7 are implemented.

Images