Multi-sensor fusion navigation method and device and storage medium

Abstract

The invention discloses a multi-sensor fusion navigation method and device and a storage medium, and the multi-sensor fusion navigation method comprises the steps: obtaining image data of visual identification code bands collected by visual sensors at two sides, calculating the pose of a mobile device in a to-be-positioned scene to obtain a visual perception pose, the distances between the two sides of the mobile device and the visual identification code band are calculated, and visual perception distances are obtained; the distance between the mobile device and the visual identification code band collected by the distance measuring sensors on the two sides is obtained, and a distance measuring sensing distance is obtained; combining the visual perception spacing with the distance measurement perception spacing to obtain a final perception spacing; and based on the final perception spacing and the visual perception pose, navigating the mobile device to move to the task end point. Comprehensive positioning of multiple sensors and multiple identification markers is achieved, the posture of the mobile equipment can be obtained more accurately by combining the visual perception distance and the distance measurement perception distance, and the navigation precision is improved.

Description

Technical Field

[0001] The present application relates to the field of device navigation technology, and in particular to a multi-sensor fusion navigation method, device and storage medium. Background Art

[0002] Mobile devices refer to robots with autonomous movement and environmental perception capabilities, such as logistics robots, automatic guided vehicles, tour guide robots and sweeping robots.

[0003] In the robot system, navigation technology is one of its most important core technologies. Navigation technology mainly determines the current position information of the robot through sensor technology, and clarifies the robot's running route based on the destination information. At present, robot navigation technology can be divided into two methods: one is to rely solely on global maps and global positioning to guide the mobile device to the target position; the other is to pre-set reference objects, and dynamically plan the navigation path through the position and posture of the reference objects relative to the mobile device. Navigation based on reference objects has higher stability, accuracy and flexibility.

[0004] However, in the process of navigation based on reference objects, it generally only supports centimeter-level path navigation, which cannot meet the requirements of high-precision tasks. Summary of the invention

[0005] In order to solve the above technical problems, the present application at least provides a multi-sensor fusion navigation method, device and storage medium.

[0006] The first aspect of the present application provides a multi-sensor fusion navigation method, the method comprising: a visual target is deployed relative to a task end point in a scene to be positioned, the visual target is composed of two visual recognition code strips respectively arranged on both sides of a driving path, and a mobile device is respectively deployed with a visual sensor and a ranging sensor on both sides, the method comprising: acquiring image data of the visual recognition code strips respectively collected by the visual sensors on both sides, and obtaining a visual coordinate system pose of the visual recognition code strip relative to the visual sensor; calculating the pose of the mobile device in the scene to be positioned based on calibration parameters of the visual sensor, the pose of the visual recognition code strip in a world coordinate system, and the visual coordinate system pose of the visual recognition code strip, and obtaining a visual perception pose; calculating the distance between the two sides of the mobile device and the visual recognition code strip respectively using the visual perception poses respectively calculated using the image data collected by the visual sensors on both sides, and obtaining a visual perception distance; and acquiring the distance between the mobile device and the visual recognition code strip collected by the ranging sensors on both sides, and obtaining a ranging perception distance; combining the visual perception distance and the ranging perception distance, obtaining a final perception distance; and navigating the mobile device to move to the task end point based on the final perception distance and the visual perception pose.

[0007] In one embodiment, the visual perception distance and the ranging perception distance are combined to obtain the final perception distance, including: calculating the difference between the visual perception distance and the ranging perception distance; if the difference between the visual perception distance and the ranging perception distance is greater than a first threshold, selecting the ranging perception distance as the final perception distance; if the difference between the visual perception distance and the ranging perception distance is not greater than the first threshold, selecting the visual perception distance as the final perception distance.

[0008] In one embodiment, the visual perception distance and the ranging perception distance are combined to obtain the final perception distance, including: based on the fluctuation trend of the visual perception distance recorded in a preset time period and the fluctuation trend of the ranging perception distance recorded in a preset time period, respectively calculating the accuracies corresponding to the visual perception distance and the ranging perception distance; and selecting the one with the highest accuracy from the visual perception distance and the ranging perception distance as the final perception distance.

[0009] In one embodiment, the visual perception distance and the ranging perception distance are combined to obtain a final perception distance, including: calculating the accuracies corresponding to the visual perception distance and the ranging perception distance based on the fluctuation trend of the visual perception distance recorded within a preset time period and the fluctuation trend of the ranging perception distance recorded within the preset time period; setting the weights of the visual perception distance and the ranging perception distance based on the accuracies corresponding to the visual perception distance and the ranging perception distance; and performing weighted summation of the visual perception distance and the ranging perception distance according to the weights of the visual perception distance and the ranging perception distance to obtain the final perception distance.

[0010] In one embodiment, before combining the visual perception interval and the ranging perception interval to obtain the final perception interval, the method further includes: taking the data collection timestamp corresponding to the visual perception interval as the reference time, selecting at least two ranging perception intervals that are adjacent to the reference time; based on the selected ranging perception interval, interpolating and calculating the ranging perception interval at the reference time to obtain the ranging perception interval after time synchronization.

[0011] In one embodiment, the final perceived distance is composed of a first distance and a second distance between the two sides of the mobile device and the visual identification code band respectively; based on the final perceived distance and the visual perception posture, the mobile device is navigated to move to the task end point, including: calculating the difference between the first distance and the second distance; if the difference between the first distance and the second distance is greater than a second threshold, then based on the difference between the first distance and the second distance, the mobile device is controlled to adjust its posture; in response to the difference between the first distance and the second distance corresponding to the mobile device after the posture adjustment being not greater than the second threshold, the mobile device is continued to be controlled to move toward the task end point based on the visual perception posture.

[0012] In one embodiment, based on the final perceived distance and the visually perceived posture, navigating the mobile device to move to the task end point includes: in response to the mobile device entering an adjacent area of ​​the task end point, detecting the size relationship between the difference between the first distance and the second distance and a third threshold; if the difference between the first distance and the second distance is greater than the third threshold, stopping the movement of the mobile device; if the difference between the first distance and the second distance is not greater than the third threshold, continuing to navigate the mobile device to move to the task end point based on the visually perceived posture.

[0013] In one embodiment, a reflective target is provided corresponding to the visual target, and a laser radar is also deployed on the mobile device; before acquiring the image data of the visual identification code band respectively acquired by the visual sensors on both sides, the method further includes: acquiring the point cloud data of the scene to be located acquired by the laser radar, and calculating the posture of the reflective target using the point cloud data; based on the posture relationship between the reflective target and the visual target, navigating the mobile device to move to the recognition area of ​​the visual target; in response to the mobile device entering the recognition area of ​​the visual target, starting the visual sensor to acquire the image data of the visual target.

[0014] According to a second aspect of the present application, there is provided a multi-sensor fusion navigation device, which includes: an image analysis module, which is used to obtain image data of a visual identification code strip respectively collected by visual sensors on both sides, and obtain a visual coordinate system pose of the visual identification code strip relative to the visual sensor; a pose calculation module, which is used to calculate the pose of a mobile device in a scene to be positioned based on calibration parameters of the visual sensor, the pose of the visual identification code strip in the world coordinate system, and the visual coordinate system pose of the visual identification code strip, and obtain a visual perception pose; a first distance calculation module, which is used to calculate the distances between the two sides of the mobile device and the visual identification code strip respectively using the visual perception poses respectively calculated using the image data collected by the visual sensors on both sides, and obtain a visual perception distance; and, obtain the distances between the mobile device and the visual identification code strip collected by the ranging sensors on both sides, and obtain a ranging perception distance; a second distance calculation module, which is used to combine the visual perception distance and the ranging perception distance to obtain a final perception distance; and a navigation module, which is used to navigate the mobile device to a task end point based on the final perception distance and the visual perception pose.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned multi-sensor fusion navigation method.

[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned multi-sensor fusion navigation method when the program instructions are executed by a processor.

[0017] The above scheme obtains the image data of the visual recognition code bands respectively collected by the visual sensors on both sides, calculates the posture of the mobile device in the scene to be positioned, obtains the visual perception posture, and calculates the distance between the two sides of the mobile device and the visual recognition code bands to obtain the visual perception distance; and obtains the distance between the mobile device and the visual recognition code band collected by the ranging sensors on both sides to obtain the ranging perception distance; combines the visual perception distance and the ranging perception distance to obtain the final perception distance; based on the final perception distance and the visual perception posture, navigates the mobile device to move to the task end point, and can combine the visual sensors deployed on both sides of the mobile device, the ranging sensors and the visual recognition code bands deployed on both sides of the driving path to realize the comprehensive positioning of multi-sensor and multi-recognition markers, and by combining the visual perception distance and the ranging perception distance, the posture of the mobile device can be more accurately known, the navigation accuracy is improved, and it can be suitable for high-precision navigation tasks.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0021] Figure 2 is a flow chart of a multi-sensor fusion navigation method shown in an exemplary embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a visual target shown in an exemplary embodiment of the present application;

[0023] Figure 4 is a schematic diagram of a visual target shown in another exemplary embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a visual target shown in another exemplary embodiment of the present application;

[0025] Figure 6 is a schematic diagram of calculating visual perception distance shown in an exemplary embodiment of the present application;

[0026] Figure 7 is a navigation schematic diagram shown in an exemplary embodiment of the present application;

[0027] Figure 8 is a flow chart of a multi-sensor fusion navigation method shown in another exemplary embodiment of the present application;

[0028] Fig. 9 is a block diagram of a multi-sensor fusion navigation device shown in an exemplary embodiment of the present application;

[0029] Fig.10 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application;

[0030] Fig.11 It is a schematic diagram of the structure of a computer-readable storage medium shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0031] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0032] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0033] The term "and / or" in this article is only an association information describing the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and next associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0034] The multi-sensor fusion navigation method provided in the embodiment of the present application is described below.

[0035] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application. Figure 1 As shown, the implementation environment includes a visual target 110, a mobile device 120 and a server 130. The visual target is composed of two visual identification code bands respectively arranged on both sides of the driving path, specifically including a visual identification code band 111 and a visual identification code band 112. The mobile device 120 and the server 130 are connected by a passage.

[0036] The visual identification code band 111 and the visual identification code band 112 contain structured annotations for visual identification, which may contain one or more visual identification codes, including but not limited to ArUco codes, QR codes, etc.

[0037] The mobile device 120 includes, but is not limited to, a logistics robot, an automatic guided vehicle (AGV), a tour guide robot, and a sweeping robot.

[0038] Visual sensors and distance sensors for detecting the visual target 110 are respectively deployed on both sides of the mobile device 120. The visual sensors and distance sensors are used to sense information such as the distance between the mobile device 120 and the visual target 110 and the posture of the mobile device 120.

[0039] Server 130 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0040] In one example, the server 130 may perform path planning based on information uploaded by the mobile device 120, such as the distance between the mobile device 120 and the visual target 110 and the position of the mobile device 120, so as to control the movement of the mobile device 120 according to the path planning result.

[0041] In one example, a client of a target application is installed and running in the mobile device 120. For example, the target application may be an application that provides a route planning function, and the route planning is implemented through the target application. The server 130 may be a background server of the target application, which is used to provide background services for the client of the target application.

[0042] In the multi-sensor fusion navigation method provided in the embodiment of the present application, the execution subject of each step can be the mobile device 120, such as the client of the target application installed and running in the mobile device 120, or the server 130, or the mobile device 120 and the server 130 can interact and cooperate to execute, that is, part of the steps of the method are executed by the mobile device 120 and the other steps are executed by the server 130.

[0043] See also Figure 2 , Figure 2 is a flowchart of a multi-sensor fusion navigation method shown in an exemplary embodiment of the present application. The multi-sensor fusion navigation method can be applied to Figure 1 It should be understood that the method can also be applied to other exemplary implementation environments and be specifically executed by devices in other implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable.

[0044] like Figure 2 As shown, the multi-sensor fusion navigation method includes at least steps S210 to S250, which are described in detail as follows:

[0045] Step S210: Acquire the image data of the visual recognition code strips respectively collected by the visual sensors on both sides to obtain the visual coordinate system position relative to the visual sensors.

[0046] The visual sensors on both sides of the mobile device collect image data of the visual recognition code band.

[0047] For example, a visual identification code strip containing multiple visual identification codes is shown in Figure 3 , Figure 3 A schematic diagram of a visual target is shown as an exemplary embodiment of the present application, such as Figure 3 As shown, the left visual identification code band and the right visual identification code band are composed of multiple ArUco codes arranged in a straight line at a certain interval, and the ArUco codes arranged in the visual identification code bands on both sides are symmetrically arranged to facilitate the subsequent visual sensors to synchronously identify the visual identification code bands on both sides.

[0048] Specifically, with a certain ArUco code (such as Figure 3 The code strip coordinate system is established with the code center of the ArUco code marked by the dotted box in the middle, so that the position and posture of the visual sensor in the code strip coordinate system can be calculated through the code identification (Identity Document, ID) corresponding to each ArUco code during identification. Because the ArUco codes of the visual identification code strips on both sides are arranged symmetrically, the coordinate systems of the code strips on both sides can be unified by taking the ArUco code with the same ID as the coordinate origin, so that the visual sensor can synchronously identify the code strips on both sides.

[0049] Of course, in addition to Figure 3 In addition to the implementation methods of the visual identification code band listed above, other forms of visual identification code bands can also be used, for example, see Figure 4 , Figure 4 A schematic diagram of a visual target is shown as another exemplary embodiment of the present application, as shown in FIG. Figure 4 As shown, a single visual identification code band can be composed of multiple rows of ArUco codes; for example, see Figure 5 , Figure 5 A schematic diagram of a visual target is shown as another exemplary embodiment of the present application, as shown in FIG. Figure 5As shown, a single visual identification code band contains a single row of ArUco codes, and the same ArUco codes are added above and below the ArUco codes at intervals of a preset number of ArUco codes in the single row of ArUco codes. Of course, visual targets in other forms of expression can also be used, such as ArUco codes arranged and combined into geometric shapes, ArUco codes arranged and combined into wave shapes, etc. This application does not limit the form of expression of the visual targets.

[0050] According to the collected image data of the visual recognition code strip, the visual coordinate system position of the visual recognition code strip relative to the visual sensor is determined.

[0051] It should be noted that if the visual identification code strip contains one visual identification code, since the size of the visual identification code is known, the image features of the visual identification code in the image data can be extracted to obtain the visual coordinate system pose of the visual identification code strip; if the visual identification code strip contains multiple visual identification codes, since the size of each visual identification code and the spacing between the visual identification codes are known, the image features corresponding to each visual identification code contained in the image data can be extracted, and all the extracted visual identification code image features can be combined to solve the visual coordinate system pose of the visual identification code strip relative to the visual sensor.

[0052] Exemplarily, the image features of the visual identification code include but are not limited to the ID corresponding to the visual identification code and the pixel coordinates of the corner points. Based on the ID corresponding to the visual identification code and the pixel coordinates of the corner points, the PnP (Perspective-n-Point) algorithm is used to calculate the visual coordinate system pose of the visual identification code relative to the visual sensor.

[0053] Step S220: Based on the calibration parameters of the visual sensor, the posture of the visual identification code tape in the world coordinate system, and the visual coordinate system posture of the visual identification code tape, the posture of the mobile device in the scene to be positioned is calculated to obtain the visual perception posture.

[0054] The image data of the visual recognition code band collected by the mobile device at different positions and in different postures are different, so the calculated visual coordinate system pose of the visual recognition code band is different. According to the calibration parameters of the visual sensor, the pose of the visual recognition code band in the world coordinate system, and the visual coordinate system pose of the visual recognition code band, the pose of the mobile device in the scene to be positioned can be converted to obtain the visual perception pose.

[0055] For example, the external calibration parameters of the visual sensor in the mobile device body coordinate system are constants, expressed as Then the relationship between the external calibration parameters, the position and posture of the visual identification code tape in the world coordinate system, the visual coordinate system position and posture of the visual identification code tape, and the position and posture of the mobile device in the scene to be positioned can be expressed as the following formula 1:

[0056]

[0057] in, is the position of the visual identification code in the world coordinate system; is the position of the mobile device in the scene to be positioned; It is the visual coordinate system pose of the visual recognition code band obtained based on image data recognition.

[0058] Due to the All other parameters are known, so they can be solved according to formula 1

[0059] Step S230: Calculate the distance between the two sides of the mobile device and the visual recognition code band respectively using the visual perception postures calculated using the image data collected by the visual sensors on both sides to obtain the visual perception distance; and obtain the distance between the mobile device and the visual recognition code band collected by the ranging sensors on both sides to obtain the ranging perception distance.

[0060] After obtaining the visual perception postures calculated from the image data collected by the visual sensors on both sides, the distances between the two sides of the mobile device and the visual recognition code bands can be calculated.

[0061] For example, see Figure 6 , Figure 6 A schematic diagram of calculating visual perception distance is shown as an exemplary embodiment of the present application, as shown in FIG. Figure 6 As shown, CL1 is the distance between the visual sensor on the left side of the mobile device and the visual identification code belt, CL2 is the distance between the visual sensor on the right side of the mobile device and the visual identification code belt, d1 is the front-to-back offset of the mobile device calculated by the left visual sensor in the left code belt coordinate system, d2 is the front-to-back offset of the mobile device calculated by the right visual sensor in the right code belt coordinate system, DL1 is the distance between the mobile device's motion center and the left visual identification code belt, DL2 is the distance between the mobile device's motion center and the right visual identification code belt, and L is the distance between the left visual sensor and the mobile device's motion center. According to the angle between the camera baseline of the visual sensor and the line connecting the origin of the code belt, b = arcsin ((d1+d2) / B), and since the calibration parameters of the visual sensor are known, the angle (d+c) and L can be calculated from the calibration parameters. Based on the principle of triangle similarity, angle b and angle c are equal, so angle d is known, and then DL1 can be calculated according to the following formula 2:

[0062] DL1=CL1+L*sin(d) (Formula 2)

[0063] Similarly, the distance DL2 between the moving center of the mobile device and the right visual identification code band can be calculated.

[0064] DL1 and DL2 are regarded as visual perception distances.

[0065] Furthermore, since the mobile device is provided with distance measuring sensors on both sides, the distance between the mobile device and the visual recognition code strips on both sides is sensed by the distance measuring sensors on both sides, thereby obtaining the distance measured and sensed.

[0066] Step S240: combining the visual perception distance and the ranging perception distance to obtain a final perception distance.

[0067] For example, a more accurate and higher perception distance may be selected from the visual perception distance and the ranging perception distance as the final perception distance.

[0068] Exemplarily, a weighted summation may be performed on the visual perception distance and the ranging perception distance to obtain a final perception distance.

[0069] Step S250: Navigate the mobile device to the task endpoint based on the final perceived distance and visually perceived posture.

[0070] For example, in the process of controlling the mobile device to move toward the task endpoint according to the visually perceived posture, the posture of the mobile device may be adjusted according to the final perceived distance.

[0071] Exemplarily, when controlling the mobile device to move toward the task end point according to the visual perception posture, it is also possible to determine whether the mobile device is in a dangerous posture according to the final perception distance. If it is in a dangerous posture, the mobile device is driven according to a preset strategy, including but not limited to reducing the moving speed, changing the moving direction, stopping on the spot, issuing an alarm signal, etc.; if it is not in a dangerous posture, the mobile device continues to move toward the task end point.

[0072] Exemplarily, the final perceived distance and visual perception posture can also be combined to plan the optimal path to reach the task end point. The optimal path can be planned based on one or more of the following conditions: shortest path distance, minimum driving energy consumption, shortest running time, and the distance between the mobile device and the visual identification code bands on both sides during driving is within a preset distance range. The mobile device is controlled to move to the task end point according to the optimal path.

[0073] For example, according to formula 1, the formula corresponding to any moving time t1 and the destination arrival time t2 can be expressed as the following formula 3 and formula 4:

[0074]

[0075] Combining Formula 3 and Formula 4, we can get Formula 5:

[0076]

[0077] In formula 5, the visual perception posture at time t1 is is a known quantity, calibration parameter is a constant, is the visual coordinate system pose of the visual recognition code band recognized at time t1, and is set to the visual coordinate system pose at the task end point at time t2 The task end point can be calculated and the mobile device can be navigated to the task end point. For the convenience of calculation, it is usually pre-set that the visual sensor of the mobile device is facing the origin of the code tape coordinate system when it stops at the task end point.

[0078] Next, some embodiments of the present application are described by way of example.

[0079] In some embodiments, a reflective target is provided corresponding to the visual target, and the mobile device is further equipped with a laser radar; before acquiring the image data of the visual identification code bands respectively collected by the visual sensors on both sides, the method further includes:

[0080] Step S211: Obtain point cloud data of the scene to be positioned collected by the laser radar, and use the point cloud data to calculate the position and posture of the reflective target.

[0081] For example, see Figure 7 , Figure 7 A navigation diagram is shown for an exemplary embodiment of the present application, such as Figure 7 As shown, a reflective target is arranged corresponding to the visual target. For example, in a device docking scenario, the reflective target can be attached to a planar structure below the docking structure, and the center of the reflective target overlaps with the center of the docking mechanism. Of course, the reflective target can also be deployed in other ways, which is not limited in the present application.

[0082] The mobile device is also deployed with a laser radar, which can be deployed on the top of the mobile device or on the front and back of the mobile device. This application does not limit this.

[0083] The mobile device first starts the lidar to collect point cloud data of the scene to be located, and uses the point cloud data to calculate the position and posture of the reflective target.

[0084] Step S212: Based on the positional relationship between the reflective target and the visual target, navigate the mobile device to move to the recognition area of ​​the visual target.

[0085] The recognition area of ​​the visual target refers to the area where the image data of the visual target can be collected.

[0086] If it is known based on the positional relationship between the reflective target and the visual target that the mobile device has not yet entered the recognition area of ​​the visual target, then continue to navigate the mobile device to move to the recognition area of ​​the visual target.

[0087] Step S213: In response to the mobile device entering the recognition area of ​​the visual target, the visual sensor is started to collect image data of the visual target.

[0088] For example, Figure 7 In the process, when the vehicle does not enter the recognition area of ​​the visual target, it is in the laser radar perception stage. After entering the recognition area of ​​the visual target, it enters the visual perception stage, starts the visual sensor to collect the image data of the visual target, and obtains the visual perception posture, and then navigates to the task end point. When entering the adjacent area of ​​the task end point ( Figure 7 After reaching the red dotted box area), it enters the multi-sensor fusion perception stage, combining multiple sensors such as ranging sensors, lidar, and visual sensors for positioning and navigation until it reaches the mission endpoint.

[0089] Among them, the ranging sensor can be activated after the mobile device enters the recognition area of ​​the visual target, or after calculating the effective visual perception posture, or after entering the adjacent area of ​​the task end point. This application does not limit this.

[0090] By first using the lidar to navigate the mobile device to the identification area of ​​the visual target, the respective advantages of lidar, visual sensors, and ranging sensors are combined to improve the accuracy of navigation task execution and the efficiency of equipment use.

[0091] In some embodiments, since the visual sensors on both sides can obtain two visual perception postures, the two visual perception postures can be weighted and summed to obtain the final visual perception posture. The weight used in the weighted summation can be preset based on experience or flexibly calculated. For example, the image quality score is performed on the image data collected by each visual sensor, and the weight corresponding to each visual perception posture is set according to the image quality score. The higher the image quality score, the higher the weight. The posture of the mobile device perceived by other sensors can also be obtained, and the difference between each visual perception posture and the posture of the mobile device perceived by other sensors is calculated. The weight corresponding to each visual perception posture is set according to the difference. The smaller the difference, the higher the weight. Of course, the weight corresponding to each visual perception posture can also be set by comprehensively considering the image quality score and the difference between the posture of the mobile device perceived by other sensors.

[0092] In some embodiments, any visual perception posture may also be selected as the final visual perception posture, such as selecting the visual perception posture calculated from the image data with the highest image quality score as the final visual perception posture; or, selecting the visual perception posture with the smallest difference between the posture of the mobile device perceived by other sensors as the final visual perception posture; or, calculating the recommendation index of each visual perception posture based on the comprehensive image quality score and the difference between the posture of the mobile device perceived by other sensors, and selecting the visual perception posture with the highest recommendation index as the final visual perception posture.

[0093] Next, various embodiments of combining the visual perception distance and the ranging perception distance to obtain the final perception distance are described by example:

[0094] Embodiment 1: In step S240, the visual perception distance and the ranging perception distance are combined to obtain the final perception distance, including:

[0095] Step S2411: Calculate the difference between the visual perception distance and the ranging perception distance.

[0096] Step S2412: Compare the size relationship between the difference between the visual perception distance and the ranging perception distance and the first threshold. If the difference between the visual perception distance and the ranging perception distance is greater than the first threshold, execute step S2413; if the difference between the visual perception distance and the ranging perception distance is not greater than the first threshold, execute step S2414.

[0097] Step S2413: Select the ranging perception distance as the final perception distance.

[0098] Step S2414: Select the visual perception distance as the final perception distance.

[0099] To facilitate calculation, the distance measuring sensor and the visual sensor can be installed adjacent to each other. When the visual sensor recognizes the visual perception distance, the distance measuring sensor can generally illuminate the visual recognition code strips on both sides. The distance measuring sensor is turned on to calculate the offset of the motion center of the mobile device on both sides of the driving path to obtain the distance measuring perception distance (the calculation method is the same as the calculation steps of the visual perception distance, and CL1 and CL2 are replaced with the distance measurement results on both sides), and cross-checked with the visual perception distance.

[0100] Specifically, if the difference between the visual perception distance and the ranging perception distance is greater than a first threshold, the ranging perception distance is preferentially used as the final perception distance, otherwise the visual perception distance is used as the final perception distance.

[0101] Of course, when the ranging perception distance is invalid (for example, because the ranging sensor is started later, the valid data may be later than the visual sensor), the visual perception distance is used as the final perception distance; if the ranging perception distance is valid but the difference between the ranging perception distance and the visual perception distance is greater than the first threshold, the ranging perception distance is used as the final perception distance.

[0102] Optionally, when the difference between the visual perception distance and the ranging perception distance is not greater than the first threshold, the visual perception distance and the ranging perception distance with the highest accuracy may be selected as the final perception distance.

[0103] Embodiment 2: In step S240, the visual perception distance and the ranging perception distance are combined to obtain the final perception distance, including:

[0104] Step S2421: based on the fluctuation trend of the visual perception distance recorded within a preset time period and the fluctuation trend of the ranging perception distance recorded within a preset time period, respectively calculate the corresponding accuracy of the visual perception distance and the ranging perception distance.

[0105] Detect whether the current visual perception distance conforms to the fluctuation trend of the visual perception distance recorded within a preset time period. If the current visual perception distance is more in line with the fluctuation trend, the accuracy is higher, and the accuracy of the visual perception distance is obtained; similarly, the accuracy corresponding to the ranging perception distance is obtained.

[0106] Step S2422: Select the one with the highest accuracy from the visual perception distance and the ranging perception distance as the final perception distance.

[0107] Optionally, in addition to calculating the accuracy corresponding to the visual perception distance based on the fluctuation trend, the accuracy corresponding to the visual perception distance can also be calculated based on one or more of the image quality score of the image data, the difference between the visual perception distance and the distance of the mobile device perceived by other sensors, the degree of matching between the visual perception postures calculated on both sides, etc., wherein the higher the image quality score, the smaller the difference, and the higher the degree of matching, the higher the accuracy of the visual perception distance.

[0108] Optionally, in addition to calculating the accuracy corresponding to the ranging perception distance based on the fluctuation trend, the accuracy corresponding to the ranging perception distance can also be calculated based on one or more of the difference between the ranging perception distance and the distance between the mobile device perceived by other sensors, the degree of matching between the ranging perception distances calculated on both sides, etc., wherein the smaller the difference and the higher the degree of matching, the higher the accuracy of the ranging perception distance.

[0109] Embodiment 3: In step S240, the visual perception distance and the ranging perception distance are combined to obtain the final perception distance, including:

[0110] Step S2431: based on the fluctuation trend of the visual perception distance recorded within a preset time period and the fluctuation trend of the ranging perception distance recorded within a preset time period, respectively calculate the corresponding accuracies of the visual perception distance and the ranging perception distance.

[0111] Step S2432: Based on the corresponding accuracies of the visual perception distance and the ranging perception distance, weights of the visual perception distance and the ranging perception distance are set respectively.

[0112] Step S2433: according to the weights of the visual perception distance and the ranging perception distance, weighted sum is performed on the visual perception distance and the ranging perception distance to obtain a final perception distance.

[0113] The higher the accuracy, the higher the weight for weighted summation.

[0114] Of course, in addition to the fluctuation trend of the visual perception distance recorded within the preset time period and the fluctuation trend of the ranging perception distance recorded within the preset time period, and the method of calculating the accuracy corresponding to the visual perception distance and the ranging perception distance respectively, you can also refer to other methods of calculating accuracy as exemplified in Example 2, and this application is not limited to this.

[0115] The final perceived distance is obtained by weighted summation of the visual perception distance and the ranging perception distance, thereby improving the accuracy of distance calculation.

[0116] In some embodiments, before combining the visual perception distance and the ranging perception distance to obtain the final perception distance, the method further includes: taking the data acquisition timestamp corresponding to the visual perception distance as the reference time, selecting at least two ranging perception distances that are adjacent to the reference time; based on the selected ranging perception distance, interpolating and calculating the ranging perception distance at the reference time to obtain the ranging perception distance after time synchronization.

[0117] That is, based on at least two ranging perception intervals adjacent to the reference time, the ranging perception interval at the reference time is predicted by interpolation, so as to achieve time synchronization between the visual perception interval and the ranging perception interval, and improve the accuracy of subsequent calculations.

[0118] Of course, the above is only the synchronization between the visual sensor and the ranging sensor data. If more other sensors are involved, the same method can be used to synchronize the perception data of other sensors.

[0119] Also, since the data acquisition frame rate of the visual sensor is generally low, the data acquisition timestamp corresponding to the visual perception interval is used as the reference time. If there are other sensors with data acquisition frame rates lower than the data acquisition frame rate of the visual sensor, the data acquisition timestamp of the sensor can be selected as the reference time.

[0120] In some embodiments, the final perceived distance is composed of a first distance and a second distance between two sides of the mobile device and the visual identification code strip respectively; in step S250, navigating the mobile device to the task end point based on the final perceived distance and the visual perception posture includes:

[0121] Step S2511: Calculate the difference between the first spacing and the second spacing.

[0122] Step S2512: If the difference between the first distance and the second distance is greater than the second threshold, the mobile device is controlled to adjust its posture based on the difference between the first distance and the second distance.

[0123] The difference between the first distance and the second distance is reduced by adjusting the posture of the mobile device.

[0124] Step S2513: In response to the difference between the first spacing and the second spacing corresponding to the mobile device after the posture adjustment being not greater than the second threshold, the mobile device is continued to be controlled to move toward the task end point based on the visually perceived posture.

[0125] In some implementations, in step S250, navigating the mobile device to move to the task end point based on the final perceived distance and the visually perceived posture includes:

[0126] Step S2521: in response to the mobile device entering the vicinity of the task endpoint, detecting the magnitude relationship between the difference between the first spacing and the second spacing and the third threshold, if the difference between the first spacing and the second spacing is greater than the third threshold, executing step S2522, if the difference between the first spacing and the second spacing is not greater than the third threshold, executing step S2523;

[0127] Step S2522: Stop the movement of the mobile device;

[0128] Step S2523: Navigate the mobile device toward the task endpoint based on visually perceived posture.

[0129] In some high-precision tasks, after entering the vicinity of the task endpoint, in order to ensure the stability and accuracy of the movement of the mobile device, if it is detected that the difference between the first spacing and the second spacing is too large, the movement of the mobile device is stopped directly, and an alarm can be further issued to avoid device damage due to task failure.

[0130] In some embodiments, when navigating a mobile device to move toward the task endpoint based on visual perception posture, the posture of the mobile device perceived by other sensors can be further referenced, such as obtaining point cloud data of a reflective target obtained by laser radar scanning, solving the posture of the mobile device, obtaining the point cloud perception posture, fusing the visual perception posture and the point cloud perception posture to obtain the final perception posture, and navigating the mobile device to move toward the task endpoint based on the final perception posture.

[0131] In some embodiments, see Figure 8 , Figure 8 is a flowchart of a multi-sensor fusion navigation method shown in another exemplary embodiment of the present application. Figure 8 As shown, including:

[0132] Step S810: acquiring point cloud data of the scene to be positioned collected by the laser radar, and calculating the position and posture of the reflective target using the point cloud data;

[0133] Step S820: Navigating the mobile device to move to the recognition area of ​​the visual target based on the positional relationship between the reflective target and the visual target;

[0134] Step S830: in response to the mobile device entering the recognition area of ​​the visual target, starting the visual sensor to collect image data of the visual target, and starting the distance measurement sensor;

[0135] Step S840: combining the visual perception distance and the ranging perception distance to obtain a final perception distance, and navigating the mobile device according to the final perception distance and the visual perception posture;

[0136] Step S850: Determine whether the vehicle enters the vicinity of the task endpoint. If yes, execute step S860; if no, continue to execute step S830;

[0137] Step S860: Determine whether the difference between the first spacing and the second spacing of the final perception spacing is greater than a threshold, if so, execute step S870, if not, execute step S880;

[0138] Step S870: dangerous parking;

[0139] Step S880: Determine whether the task end point has been reached, if so, the task ends, if not, continue to execute step S830.

[0140] The multi-sensor fusion navigation method provided by the present application obtains image data of the visual recognition code bands respectively collected by the visual sensors on both sides, calculates the posture of the mobile device in the scene to be positioned, obtains the visual perception posture, and calculates the distance between the two sides of the mobile device and the visual recognition code bands to obtain the visual perception distance; and obtains the distance between the mobile device and the visual recognition code band collected by the ranging sensors on both sides to obtain the ranging perception distance; combines the visual perception distance and the ranging perception distance to obtain the final perception distance; based on the final perception distance and the visual perception posture, navigates the mobile device to move to the task end point, and can combine the visual sensors deployed on both sides of the mobile device, the ranging sensors and the visual recognition code bands deployed on both sides of the driving path to realize the comprehensive positioning of multi-sensor and multi-recognition markers, and by combining the visual perception distance and the ranging perception distance, the posture of the mobile device can be more accurately known, the navigation accuracy is improved, and it can be suitable for high-precision navigation tasks.

[0141] Fig. 9 FIG. 1 is a block diagram of a multi-sensor fusion navigation device shown in an exemplary embodiment of the present application. Fig. 9 As shown, the exemplary multi-sensor fusion navigation device 900 includes: an image analysis module 910, a posture calculation module 920, a first distance calculation module 930, a second distance calculation module 940 and a navigation module 950, specifically including:

[0142] The image analysis module 910 is used to obtain the image data of the visual identification code strips respectively collected by the visual sensors on both sides, and obtain the visual coordinate system pose of the visual identification code strips relative to the visual sensors;

[0143] A posture calculation module 920 is used to calculate the posture of the mobile device in the scene to be positioned based on the calibration parameters of the visual sensor, the posture of the visual recognition code tape in the world coordinate system, and the visual coordinate system posture of the visual recognition code tape to obtain the visual perception posture;

[0144] The first distance calculation module 930 is used to calculate the distances between the two sides of the mobile device and the visual recognition code strips respectively using the visual perception postures respectively calculated by the image data collected by the visual sensors on both sides to obtain the visual perception distance; and obtain the distances between the mobile device and the visual recognition code strips collected by the ranging sensors on both sides to obtain the ranging perception distance;

[0145] A second distance calculation module 940 is used to combine the visual perception distance and the ranging perception distance to obtain a final perception distance;

[0146] The navigation module 950 is used to navigate the mobile device to the task end point based on the final perceived distance and the visually perceived posture.

[0147] It should be noted that the multi-sensor fusion navigation device provided in the above embodiment and the multi-sensor fusion navigation method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the multi-sensor fusion navigation device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0148] See also Fig.10 , Fig.10 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 1000 includes a memory 1001 and a processor 1002, and the processor 1002 is used to execute program instructions stored in the memory 1001 to implement the steps in any of the above multi-sensor fusion navigation method embodiments. In a specific implementation scenario, the electronic device 1000 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 1000 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0149] Specifically, the processor 1002 is used to control itself and the memory 1001 to implement the steps in any of the above-mentioned multi-sensor fusion navigation method embodiments. The processor 1002 can also be called a central processing unit (CPU). The processor 1002 may be an integrated circuit chip with signal processing capabilities. The processor 1002 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 1002 can be implemented by an integrated circuit chip.

[0150] See also Fig.11 , Fig.11 The computer-readable storage medium 1100 stores program instructions 1110 that can be executed by a processor, and the program instructions 1110 are used to implement the steps in any of the above multi-sensor fusion navigation method embodiments.

[0151] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0152] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0153] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0154] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program code.

Claims

1. A multi-sensor fusion navigation method, characterized in that: A visual target is deployed relative to the task end point in the scene to be positioned, and the visual target is composed of two visual identification code bands respectively arranged on both sides of the driving path. The mobile device is respectively deployed with a visual sensor and a distance measurement sensor on both sides. The method includes: Acquire image data of the visual recognition code strip respectively collected by the visual sensors on both sides, and obtain the visual coordinate system pose of the visual recognition code strip relative to the visual sensor; Based on the calibration parameters of the visual sensor, the pose of the visual identification code band in the world coordinate system, and the pose of the visual coordinate system of the visual identification code band, the pose of the mobile device in the scene to be positioned is calculated to obtain a visual perception pose; Calculate the distances between the two sides of the mobile device and the visual recognition code strip respectively using the visual perception postures respectively calculated using the image data collected by the visual sensors on both sides to obtain the visual perception distance; and obtain the distances between the mobile device and the visual recognition code strip collected by the ranging sensors on both sides to obtain the ranging perception distance; Combining the visual perception distance and the ranging perception distance to obtain a final perception distance; Based on the final perceived distance and the visually perceived posture, the mobile device is navigated to move to the task end point.

2. The method according to claim 1, characterized in that The combining the visual perception distance and the ranging perception distance to obtain a final perception distance includes: Calculating the difference between the visual perception distance and the ranging perception distance; If the difference between the visual perception distance and the ranging perception distance is greater than a first threshold, selecting the ranging perception distance as the final perception distance; If the difference between the visual perception distance and the ranging perception distance is not greater than the first threshold, the visual perception distance is selected as the final perception distance.

3. The method according to claim 1, characterized in that: The combining the visual perception distance and the ranging perception distance to obtain a final perception distance includes: Based on the fluctuation trend of the visual perception distance recorded within a preset time period and the fluctuation trend of the ranging perception distance recorded within a preset time period, respectively calculating the corresponding accuracy of the visual perception distance and the ranging perception distance; The one with the highest accuracy is selected from the visual perception distance and the ranging perception distance as the final perception distance.

4. The method according to claim 1, characterized in that The combining the visual perception distance and the ranging perception distance to obtain a final perception distance includes: Based on the fluctuation trend of the visual perception distance recorded within a preset time period and the fluctuation trend of the ranging perception distance recorded within a preset time period, respectively calculating the corresponding accuracy of the visual perception distance and the ranging perception distance; Based on the corresponding accuracies of the visual perception distance and the ranging perception distance, respectively, weights of the visual perception distance and the ranging perception distance are set; According to the weights of the visual perception distance and the ranging perception distance, the visual perception distance and the ranging perception distance are weightedly summed to obtain a final perception distance.

5. The method according to any one of claims 1 to 4, characterized in that: Before combining the visual perception distance and the ranging perception distance to obtain a final perception distance, the method further includes: Taking the data collection timestamp corresponding to the visual perception interval as a reference time, selecting at least two ranging perception intervals that are adjacent to and before the reference time; Based on the selected ranging perception interval, the ranging perception interval at the reference time is interpolated and calculated to obtain the ranging perception interval after time synchronization.

6. The method according to claim 1, characterized in that The final perception distance is composed of a first distance and a second distance between two sides of the mobile device and the visual identification code band respectively; and navigating the mobile device to the task end point based on the final perception distance and the visual perception posture includes: calculating a difference between the first spacing and the second spacing; If the difference between the first spacing and the second spacing is greater than a second threshold, controlling the mobile device to adjust its posture based on the difference between the first spacing and the second spacing; In response to the difference between the first spacing and the second spacing corresponding to the mobile device after posture adjustment being not greater than the second threshold, the mobile device is continuously controlled to move toward the task endpoint based on the visually perceived posture.

7. The method according to claim 1, characterized in that The final perception distance is composed of a first distance and a second distance between two sides of the mobile device and the visual identification code band respectively; and navigating the mobile device to the task end point based on the final perception distance and the visual perception posture includes: In response to the mobile device entering a neighborhood of the task endpoint, detecting a magnitude relationship between a difference between the first spacing and the second spacing and a third threshold; If the difference between the first distance and the second distance is greater than the third threshold, stopping the movement of the mobile device; If the difference between the first distance and the second distance is not greater than the third threshold, continue to navigate the mobile device toward the task endpoint based on the visual perception posture.

8. The method according to claim 1, characterized in that: A reflective target is arranged corresponding to the visual target, and the mobile device is also equipped with a laser radar; before acquiring the image data of the visual identification code band respectively collected by the visual sensors on both sides, the method further includes: Acquire point cloud data of the scene to be positioned collected by the laser radar, and calculate the position and posture of the reflective target using the point cloud data; Based on the positional relationship between the reflective target and the visual target, navigating the mobile device to move to the recognition area of ​​the visual target; In response to the mobile device entering the recognition area of ​​the visual target, the visual sensor is started to collect image data of the visual target.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the steps in the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and the program instructions can be executed by a processor to implement the steps in the method according to any one of claims 1 to 8.

Images