Indoor dynamic environment mapping method and system based on deep learning

Abstract

The invention provides an indoor dynamic environment mapping method and system based on deep learning, and the method comprises the steps: building an indoor dynamic environment mapping model based on a deep learning technology; creating a mapping demand; and after the creation is completed, inputting a mapping demand into the indoor dynamic environment mapping model, and generating a mapping result. According to the indoor dynamic environment mapping method and system based on deep learning, an improved lightweight framework algorithm is provided to improve YOLOv5, and a lightweight network meeting real-time detection conditions is trained and selected. And then designing a framework of VSLAM and ORB-SLAM3 in a dynamic environment, taking overlapping of a dynamic frame and a static frame as a judgment mechanism, combining with an improved multi-view geometric dynamic region detection module, and training to obtain an indoor dynamic environment mapping model. And deep learning is carried out on a scanning identification result to reserve dynamic object feature points, so that the robustness and accuracy of mapping are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and particularly relates to a method and system for indoor dynamic environment mapping based on deep learning. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) technology is a technology that allows a robot or an autonomous driving vehicle to navigate autonomously in an unknown environment. It solves two problems simultaneously: determining the position of the robot in the environment (localization) and constructing a map of the environment (mapping). The core principle of SLAM technology lies in estimating the position of the robot and the environmental map simultaneously without prior knowledge of the environment where the robot is located. This process involves continuously updating the position estimate of the robot and map construction. Currently, most of the Simultaneous Localization and Mapping (SLAM) methods are for static environments, and intelligent vehicles have poor positioning in dynamic environments, with defects such as blurring and afterimages.

[0003] In view of this, there is an urgent need for a method and system for indoor dynamic environment mapping based on deep learning to at least solve the above deficiencies. Summary of the Invention

[0004] One of the objectives of the present invention is to provide a method and system for indoor dynamic environment mapping based on deep learning, propose an improved lightweight framework algorithm and improve YOLOv5, and then through the training and testing of the COCO dataset and indoor dataset, select a lightweight network that meets the real-time detection conditions. Then design the frameworks of VSLAM and ORB-SLAM3 in a dynamic environment to adapt to the dynamic environment. At the same time, use an improved multi-view geometry dynamic region detection module with the overlapping of static and dynamic frames as the judgment mechanism. Finally, train the algorithm effect after fusing the two to obtain an indoor dynamic environment mapping model. The model processes the mapping requirements, avoids defects such as blurring and afterimages in a dynamic environment, and greatly improves the robustness and accuracy of mapping.

[0005] A method for indoor dynamic environment mapping based on deep learning provided by an embodiment of the present invention includes:

[0006] Based on deep learning technology, construct an indoor dynamic environment mapping model;

[0007] Create mapping requirements;

[0008] After creation, input the mapping requirements into the indoor dynamic environment mapping model to generate a mapping result.

[0009] Preferably, based on deep learning technology, constructing an indoor dynamic environment mapping model includes:

[0010] Lightweight improvement is made to the backbone network of YOLOv5 based on the PP-LCNet module to obtain improved YOLOv5;

[0011] Train and test the improved YOLOv5 using the COCO dataset and the indoor dataset, and select a lightweight network that meets the real-time detection conditions;

[0012] Design the frameworks of VSLAM and ORB-SLAM3 in a dynamic environment based on the lightweight network to adapt to the dynamic environment, and use the overlap of static and dynamic frames as the judgment mechanism;

[0013] Based on the judgment mechanism and the improved multi-view geometry dynamic region detection module, determine the mapping model for the indoor dynamic environment.

[0014] Preferably, create a mapping requirement, including:

[0015] Obtain the tracking dialogue information of the mapping requirement space;

[0016] According to the tracking dialogue information, determine at least one mapping requirement semantics;

[0017] According to the mapping requirement semantics, instruct the target person to select the mapping area;

[0018] Plan the real-time cruise trajectory of the intelligent vehicle in the mapping area, trigger the intelligent vehicle to cruise based on the real-time cruise trajectory, and use the first lidar scan data of the intelligent vehicle during the cruise as the mapping requirement.

[0019] Preferably, planning the real-time cruise trajectory of the intelligent vehicle in the mapping area includes:

[0020] Obtain the real-time scan data of the intelligent vehicle and determine the target obstacles during the real-time scan;

[0021] Obtain the motion state of the target obstacle; wherein, the motion state includes: moving state and stationary state;

[0022] Plan the bypass route of the target obstacle and control the intelligent vehicle to bypass based on the bypass route;

[0023] After bypassing, according to the different motion states, perform route planning and control the intelligent vehicle to drive when the intelligent vehicle recognizes the bypass position next time, and obtain the local mapping result within the preset range of the bypass position;

[0024] Perform bypass standardization detection according to the local mapping result, and control the intelligent vehicle to supplement the scan when the bypass standardization detection result fails.

[0025] Preferably, performing bypass standardization detection according to the local mapping result and controlling the intelligent vehicle to supplement the scan when the bypass standardization detection result fails includes:

[0026] If the motion state is the moving state, attempt to obtain the bottom map gap in the local mapping result. If the attempt to obtain is successful, it is determined that the bypass standardization detection result fails. Analyze the target position of the bottom map gap in the local mapping result, and control the intelligent vehicle to supplement the scanning based on the target position.

[0027] If the motion state is the stationary state, obtain the gap contour of the bottom map gap in the local mapping result.

[0028] Obtain the placement bottom contour of the target obstacle.

[0029] Perform contour matching on the gap contour and the placement bottom contour. If the contour matching is compliant, the bypass standardization detection result passes; otherwise, it fails, and supplementary scanning is performed based on the contour matching difference points.

[0030] Preferably, performing supplementary scanning based on the contour matching difference points includes:

[0031] Project the intelligent vehicle and the placement bottom contour onto the local mapping result to obtain a reference three-dimensional model.

[0032] Determine the difference contour points between the gap contour and the placement bottom contour in the reference three-dimensional model, connect the difference contour points and the position of the laser camera, and determine the auxiliary line.

[0033] Determine the virtual camera area of the laser camera in the reference three-dimensional model.

[0034] Characterize the positional relationship between the auxiliary line and the virtual camera area to obtain a position feature set.

[0035] Determine the intelligent vehicle attitude control instruction according to the position feature set and the preset intelligent vehicle attitude control instruction library.

[0036] Execute the intelligent vehicle attitude control instruction. When the adjusted position feature set conforms to the standard position feature, control the intelligent vehicle laser scanning.

[0037] An indoor dynamic environment mapping method based on deep learning provided by an embodiment of the present invention further includes:

[0038] Analyze the mapping result, determine the environmental target, and supplement the mapping according to the environmental target and the line-of-sight information of the target person.

[0039] Preferably, supplementing the mapping according to the environmental target and the line-of-sight information of the target person includes:

[0040] Determine the line-of-sight following target of the target person according to the environmental target and the line-of-sight information of the target person.

[0041] Obtain the target direction vector of the line-of-sight following target.

[0042] Determine the following indication direction vector of the target direction vector.

[0043] Based on the following indication direction vector, control the intelligent vehicle to track the line-of-sight following target and obtain the second lidar data during the tracking process;

[0044] Supplement the mapping according to the second lidar data and the indoor dynamic environment mapping model;

[0045] Among them, the determination conditions for the following indication direction vector for determining the target direction vector include:

[0046] The intelligent vehicle can detect the line-of-sight following target based on the vector starting point position of the following indication direction vector;

[0047] And,

[0048] The position distance between the vector starting point position of the following indication direction vector and the line-of-sight following target position is greater than or equal to a preset position distance threshold;

[0049] And,

[0050] When the position distance between the vector starting point position of the following indication direction vector and the line-of-sight following target position is greater than the preset position distance threshold, the vector included angle between the following indication direction vector and the target direction vector is less than the preset vector included angle threshold;

[0051] When the position distance between the vector starting point position of the following indication direction vector and the line-of-sight following target position is equal to the position distance threshold, the vector included angle range between the following indication direction vector and the target direction vector is linearly distributed within a preset range interval.

[0052] An indoor dynamic environment mapping system based on deep learning provided by an embodiment of the present invention includes:

[0053] A mapping model construction subsystem, configured to construct an indoor dynamic environment mapping model based on deep learning technology;

[0054] A mapping requirement creation subsystem, configured to instruct a target person to create a mapping requirement;

[0055] A mapping result generation subsystem, configured to, after creation is completed, input the mapping requirement into the indoor dynamic environment mapping model to generate a mapping result.

[0056] Preferably, the mapping model construction subsystem constructs an indoor dynamic environment mapping model based on deep learning technology and performs the following operations:

[0057] Lightweight improvement is performed on the backbone network of YOLOv5 based on the PP-LCNet module to obtain improved YOLOv5;

[0058] Train and test the improved YOLOv5 using the COCO dataset and the indoor dataset, and select a lightweight network that meets the real-time detection conditions;

[0059] Design a framework for VSLAM and ORB-SLAM3 in a dynamic environment based on the lightweight network to adapt to the dynamic environment, and use the overlap of static and dynamic frames as the judgment mechanism;

[0060] Based on the judgment mechanism and the improved multi-view geometry dynamic region detection module, determine the mapping model for the indoor dynamic environment.

[0061] Preferably, the mapping requirement creation subsystem creates mapping requirements and performs the following operations:

[0062] Obtain the tracking dialogue information of the mapping requirement space;

[0063] According to the tracking dialogue information, determine at least one mapping requirement semantics;

[0064] According to the mapping requirement semantics, instruct the target person to select the mapping area;

[0065] Plan the real-time cruise trajectory of the intelligent vehicle in the mapping area, trigger the intelligent vehicle to cruise based on the real-time cruise trajectory, and use the first lidar scan data of the intelligent vehicle during the cruise as the mapping requirement.

[0066] An indoor dynamic environment mapping system provided by an embodiment of the present invention also performs the following operations:

[0067] The mapping supplement subsystem analyzes the mapping result, determines the environmental target, and supplements the mapping according to the environmental target and the line-of-sight information of the target person.

[0068] Preferably, the mapping supplement subsystem supplements the mapping according to the environmental target and the line-of-sight information of the target person, including:

[0069] According to the environmental target and the line-of-sight information of the target person, determine the line-of-sight following target of the target person;

[0070] Obtain the target direction vector of the line-of-sight following target;

[0071] Determine the following indication direction vector of the target direction vector;

[0072] Based on the following indication direction vector, control the intelligent vehicle to track the line-of-sight following target and obtain the second lidar data during the tracking process;

[0073] Supplement the mapping according to the second lidar data and the indoor dynamic environment mapping model;

[0074] Among them, the determination conditions for determining the following indication direction vector of the target direction vector include:

[0075] The intelligent vehicle can detect the line-of-sight following target based on the vector starting position of the following indication direction vector;

[0076] Moreover,

[0077] The position distance between the vector starting position of the following indication direction vector and the line-of-sight following target position is greater than or equal to a preset position distance threshold;

[0078] Moreover,

[0079] When the position distance between the vector starting position of the following indication direction vector and the line-of-sight following target position is greater than the preset position distance threshold, the vector angle between the following indication direction vector and the target direction vector is less than the preset vector angle threshold;

[0080] When the position distance between the vector starting position of the following indication direction vector and the line-of-sight following target position is equal to the position distance threshold, the vector angle between the following indication direction vector and the target direction vector is linearly distributed within a preset range interval.

[0081] The beneficial effects of the present invention are as follows:

[0082] The present invention proposes an improved lightweight framework algorithm and improves YOLOv5. Then, through the training and testing of the COCO dataset and the indoor dataset, a lightweight network that meets the real-time detection conditions is selected. Then, a framework for VSLAM and ORB-SLAM3 in a dynamic environment is designed to adapt to the dynamic environment. At the same time, with the overlapping of static and dynamic frames as the judgment mechanism, an improved multi-view geometry dynamic region detection module is used. Finally, the algorithm effect after fusing the two is trained to obtain an indoor dynamic environment mapping model. The model processes the mapping requirements, avoids defects such as blurring and afterimages in the dynamic environment, and greatly improves the robustness and accuracy of mapping.

[0083] Other features and advantages of the present invention will be described in the following specification, and, in part, will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention can be realized and obtained through the structures specifically pointed out in this application document.

[0084] The technical solutions of the present invention will be further described in detail below through the accompanying drawings and embodiments. Description of the Drawings

[0085] The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation to the present invention. In the accompanying drawings:

[0086] Figure 1 It is a schematic diagram of a method for mapping an indoor dynamic environment based on deep learning in an embodiment of the present invention;

[0087] Figure 2 This is a schematic diagram of a deep - learning - based indoor dynamic environment mapping system in an embodiment of the present invention. Detailed implementation manners

[0088] The following describes the preferred embodiments of the present invention with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0089] An embodiment of the present invention provides a deep - learning - based indoor dynamic environment mapping method, as Figure 1 shown, including:

[0090] Step 1: Based on deep - learning technology, construct an indoor dynamic environment mapping model; wherein, the indoor dynamic environment mapping model is an AI model that can adapt to dynamic changes (such as moving objects, changing lighting, etc.) in the indoor environment and perform real - time positioning and map construction;

[0091] Step 2: Create a mapping requirement; wherein, the mapping requirement is the three - dimensional scan data of a preset lidar of an intelligent vehicle for indoor mapping. When creating, send a preset full - house scan command to the intelligent vehicle;

[0092] After the creation is completed, input the mapping requirement into the indoor dynamic environment mapping model to generate a mapping result; wherein, the mapping result is an environmental three - dimensional model corresponding to the mapping requirement;

[0093] Among them, Step 1: Based on deep - learning technology, construct an indoor dynamic environment mapping model, including:

[0094] Lightweight improvement of the backbone network of YOLOv5 based on the PP - LCNet module to obtain improved YOLOv5;

[0095] Train and test the improved YOLOv5 with the COCO dataset and indoor datasets, and select a lightweight network that meets the real - time detection conditions;

[0096] Design the frameworks of VSLAM and ORB - SLAM3 in a dynamic environment based on the lightweight network to adapt to the dynamic environment, and use the overlapping of static and dynamic frames as a judgment mechanism;

[0097] Based on the judgment mechanism and the improved multi - perspective geometry dynamic region detection module, determine the indoor dynamic environment mapping model.

[0098] The working principle and beneficial effects of the above - mentioned technical solutions are:

[0099] Indoor dynamic environment mapping based on deep learning mainly includes a ROS host computer, a lidar, and an IMU inertial measurement component. The indoor dynamic environment mapping model is an AI model for dynamic environment mapping based on lidar scan data. During construction, the PP-LCNet module is used to lightweight improve the backbone network of YOLOv5 and a self-made dataset is used for training. Aiming at the problem of poor positioning accuracy of traditional SLAM in dynamic environments, a moving and static frame mechanism and an improved multi-view geometry fusion dynamic region detection algorithm are proposed: a new target detection thread is added, and indoor objects are divided into three categories: dynamic, static, and potentially moving objects. YOLOv5s-PP is used to perform target detection on the input image and establish semantic labels; the movement priority of people is set to the highest. However, in an indoor environment, the space is relatively compact. When people move, they always pass by many static objects or when people are close to the observation point, the detection frame of people will occupy a large part of the picture. An improved YOLOv5 (a new feature pyramid network is proposed. Through adaptive feature fusion and receptive field enhancement, channel information is largely retained during feature transfer, and different receptive fields in each feature map are adaptively learned to enhance the representation of the feature pyramid, effectively improving the accuracy of multi-scale target recognition; a new automatic learning data augmentation strategy is proposed. Inspired by AutoAugment, the latest data augmentation operations are added. The improved data augmentation method effectively improves the model training effect and the robustness of the trained model.) is used to frame multiple objects, where: people, as a very active individual, are set as dynamic objects, while the screen is an absolute static object, that is, an object that is not easily moved. Therefore, the feature points within the frame of this type are always retained. Indoor objects such as backpacks are set as potentially moving objects. Aiming at the defect problems of traditional point cloud mapping, such as blurring and ghosting due to dynamic objects, during the mapping stage, a strategy of dynamic detection frames combined with geometric vision is used to remove image information. According to the point cloud of the key frames in the system, point cloud stitching of the corresponding poses is performed, and dense point cloud maps with static backgrounds are constructed for the TUM high-dynamic dataset, actual simple and complex scenarios respectively.

[0100] The present invention proposes an improved lightweight framework algorithm and improves YOLOv5. Then, through the training and testing of the COCO dataset and the indoor dataset, a lightweight network that meets the real-time detection conditions is selected. Then, a framework of VSLAM and ORB-SLAM3 in a dynamic environment is designed to adapt to the dynamic environment. At the same time, with the overlapping of the moving and static frames as the judgment mechanism, an improved multi-view geometry dynamic region detection module is used. Finally, the algorithm effect after fusing the two is trained to obtain an indoor dynamic environment mapping model. The model processes the mapping requirements, avoids defects such as blurring and ghosting in dynamic environments, and greatly improves the robustness and accuracy of mapping.

[0101] In one embodiment, creating a mapping requirement includes:

[0102] Obtain the tracking dialogue information of the mapping requirement space; wherein, the mapping requirement space is the space where the mapping requirement is issued, such as: balcony; the tracking dialogue information is the voice information of the target person collected by the intelligent vehicle, such as: "The sweeping robot needs to map the home for the first time. Please scan the home first.", and the target person is the person with the mapping requirement, such as: the user who uses the sweeping robot for the first time;

[0103] According to the tracking dialogue information, determine at least one mapping requirement semantics; wherein, the mapping requirement semantics is the semantics representing the existence of the mapping requirement, such as: "Need to map the home", "Scan the home";

[0104] According to the mapping requirement semantics, instruct the target person to select the mapping area; wherein, instructing the target person to select the mapping area means that the intelligent vehicle will map based on the space it can reach at its current position. Before mapping, the intelligent vehicle will issue a reminder: "The mapping is about to start. Please confirm to start." After the reminder, the target person needs to judge whether the intelligent vehicle is currently in the space where mapping is required. For example, if the target person wants to map the second floor of the home and the intelligent vehicle is on the first floor, it needs to be placed on the second floor before confirming to start. If the intelligent vehicle is on the first floor, it can be directly confirmed to start. The area that the intelligent vehicle on the second floor can reach is the mapping area;

[0105] Plan the real-time cruising trajectory of the intelligent vehicle in the mapping area, trigger the intelligent vehicle to cruise based on the real-time cruising trajectory, and use the first lidar scan data of the intelligent vehicle during the cruising process as the mapping requirement.

[0106] The working principle and beneficial effects of the above technical solution are:

[0107] The present invention introduces semantic analysis technology, parses the tracking dialogue information to obtain the mapping requirement semantics, according to the mapping requirement semantics, instructs the target person to select the mapping area to avoid mis-triggering, and based on the intelligent obstacle avoidance technology, plans the real-time cruising trajectory of the intelligent vehicle in the mapping area in real time, triggers the intelligent vehicle to cruise based on the real-time cruising trajectory, and uses the first lidar scan data of the intelligent vehicle during the cruising process as the mapping requirement, improving the comprehensiveness and rationality of obtaining the mapping requirement.

[0108] In one embodiment, planning the real-time cruising trajectory of the intelligent vehicle in the mapping area includes:

[0109] Obtain the real-time scan data of the intelligent vehicle and determine the target obstacles during the real-time scanning process; wherein, the target obstacles are the obstacles that block the original planned route of the intelligent vehicle during the mapping process of the intelligent vehicle. When determining, match the three-dimensional model of the obstacle with the preset model library to determine the type of the obstacle. The preset model library includes three-dimensional models of various preset obstacles, such as: the three-dimensional model of a table leg, the human body model, etc.;

[0110] Obtain the motion state of the target obstacle; wherein, the motion state includes: moving state and stationary state;

[0111] Plan a detour route for the target obstacle and control the intelligent vehicle to detour based on the detour route; wherein, the detour route is obtained based on obstacle avoidance technology;

[0112] After detouring, according to the different motion states, when the intelligent vehicle recognizes the detour position next time, perform route planning and control the intelligent vehicle to drive, and obtain the local mapping result within the preset range of the detour position; wherein, according to the different motion states, when the intelligent vehicle recognizes the detour position next time, perform route planning and control the intelligent vehicle to drive specifically as: when the motion state is stationary, re-plan to avoid the target obstacle based on obstacle avoidance technology at the detour position; when the motion state is moving, perform a scan of the detour position when the intelligent vehicle recognizes the detour position next time; the preset range is: within a circular area with the detour position as the center and the longest intercepted line segment of the ground projection contour of the target obstacle as the radius;

[0113] Perform a detour standardization detection based on the local mapping result. When the detour standardization detection result fails, control the intelligent vehicle to supplement the scan. Among them, the detour standardization detection is: judging whether there is an omission in the scanned area during the detour process of the intelligent vehicle.

[0114] The working principle and beneficial effects of the above technical solution are:

[0115] The present invention identifies the target obstacle in the real-time scan mapping process, plans a detour route based on the motion state of the target obstacle. After detouring, when the intelligent vehicle recognizes the detour position next time, perform route planning and control the intelligent vehicle to drive, and obtain the local mapping result including the detour position. Perform a detour standardization detection based on the local mapping result, judge whether the local mapping is standard. When it is not standard, control the intelligent vehicle to supplement the scan, improving the accuracy of mapping.

[0116] In one embodiment, performing a detour standardization detection based on the local mapping result. When the detour standardization detection result fails, controlling the intelligent vehicle to supplement the scan includes:

[0117] If the motion state is the moving state, try to obtain the base map gap in the local mapping result. If the attempt to obtain is successful, it is determined that the detour standardization detection result fails, analyze the target position of the base map gap in the local mapping result, and control the intelligent vehicle to supplement the scan based on the target position; wherein, the base map gap is: the area without scan information corresponding to the mapping position;

[0118] If the motion state is the stationary state, obtain the gap contour of the base map gap in the local mapping result;

[0119] Obtain the placement bottom surface contour of the target obstacle;

[0120] Perform contour matching on the notch contour and the placement bottom surface contour. If the contour matching is successful, the bypass standard detection result passes; otherwise, it fails. Based on the different points in the contour matching, perform supplementary scanning.

[0121] The working principle and beneficial effects of the above technical solution are as follows:

[0122] During the process of the intelligent vehicle building a map, when there are obstacles on its planned route, it will automatically avoid them. Since the avoidance is triggered by sudden events and the intelligent vehicle will prioritize ensuring driving safety at present and then consider map building, it may cause omissions in the map scanning data and lead to inaccurate map building. Therefore, determine the target obstacles during the real-time scanning process, introduce the motion state of the target obstacles. When the target obstacle is moving, it means that the map position currently occupied by the target obstacle can be scanned next time; otherwise, it cannot. Plan the bypass route of the target obstacle, and perform reconstruction based on the lidar data collected jointly on the bypass route and the obstacle avoidance route when the intelligent vehicle identifies the bypass position next time. When there are map gaps in the reconstructed local map, it means that the intelligent vehicle has scanning omissions and needs to perform supplementary scanning. When the target obstacle is stationary, it means that the map positions currently occupied by the target obstacle cannot be scanned, but the bottom contour of the target obstacle mapped on the ground is fixed. When the notch contour and the bottom contour match, it means that the scanning is sufficient and meets the specifications; otherwise, perform supplementary scanning based on the different points in the contour matching to improve the suitability of the supplementary scanning.

[0123] In one embodiment, performing supplementary scanning based on the different points in the contour matching includes:

[0124] Project the intelligent vehicle and the placement bottom surface contour into the local map result to obtain a reference 3D model;

[0125] Determine the different contour points between the notch contour and the placement bottom surface contour in the reference 3D model, connect the different contour points and the position of the laser camera to determine the auxiliary line; among them, the different contour points are: the points on the local placement bottom surface contour when the notch contour and the placement bottom surface contour do not match;

[0126] Determine the virtual camera area of the laser camera in the reference 3D model;

[0127] Characterize the positional relationship between the auxiliary line and the virtual camera area to obtain a position feature set; among them, the position features include: the relative positional relationship between each auxiliary line and the virtual camera area, for example: auxiliary lines 1 - 3 are not within the virtual camera area, auxiliary lines 4 - 7 intersect with the boundary of the virtual camera area, and auxiliary lines 8 - 10 are within the virtual camera area;

[0128] Determine the intelligent vehicle attitude control instruction according to the position feature set and the preset intelligent vehicle attitude control instruction library; wherein, the preset intelligent vehicle attitude control instruction library includes a preselected position feature set and an intelligent vehicle attitude control instruction in one-to-one correspondence. For example, if the preselected position feature set is that auxiliary lines 1-3 are not within the virtual camera area, auxiliary lines 4-7 intersect with the boundary of the virtual camera area, and auxiliary lines 8-10 are within the virtual camera area, then the intelligent vehicle attitude control instruction is: Rotate 30° clockwise in place;

[0129] Execute the intelligent vehicle attitude control instruction. When the adjusted position feature set conforms to the standard position feature, control the intelligent vehicle laser scanning. Among them, the adjusted position feature set is: a set of position features of the auxiliary lines and the virtual camera area extracted after the intelligent vehicle adjusts its body attitude based on the intelligent vehicle attitude control instruction; the standard position feature is: each auxiliary line is within the virtual camera area.

[0130] The working principle and beneficial effects of the above technical solution are:

[0131] When the present invention performs supplementary scanning based on the dissimilar points of contour matching, both the intelligent vehicle and the bottom contour of the placement are projected into the local mapping result to obtain a reference three-dimensional model. Determine the different contour points in the reference three-dimensional model, connect the different contour points and the position of the laser camera to determine the auxiliary lines. At the same time, obtain the virtual camera area of the laser camera (for example: a fan-shaped area with the position of the laser camera as the fan-shaped vertex); extract the position features of the auxiliary lines and the virtual camera area to obtain a position feature set, introduce the intelligent vehicle attitude control instruction library, determine the intelligent vehicle attitude control instruction to control the intelligent vehicle to adjust its attitude. When the adjusted position feature set obtained in real time conforms to the standard position feature, control the intelligent vehicle laser scanning for targeted supplementary scanning, reducing the energy consumption of the lidar.

[0132] The embodiment of the present invention provides a method for indoor dynamic environment mapping based on deep learning, which further includes:

[0133] Analyze the mapping result to determine the environmental target, and supplement the mapping according to the environmental target and the line-of-sight information of the target person. Among them, the environmental target is: a three-dimensional human body or object model scanned during the mapping process.

[0134] The working principle and beneficial effects of the above technical solution are:

[0135] The present invention introduces the line-of-sight information of the target person to supplement the mapping of the environmental target, improving the pertinence of the environmental mapping.

[0136] In one embodiment, supplementing the mapping according to the environmental target and the line-of-sight information of the target person includes:

[0137] According to the line of sight information of the environmental target and the target person, the line of sight tracking target of the target person is determined; wherein, when the line of sight stays for a time greater than a preset line of sight stay time threshold, the corresponding environmental target is the line of sight tracking target;

[0138] Obtaining a target direction vector of the line of sight tracking target; wherein the target direction vector is: a vector constructed based on the position and moving speed direction of the line of sight tracking target;

[0139] Determine a tracking indication direction vector of the target direction vector; wherein the tracking indication direction vector is: a tracking direction indication vector of the smart car near the line of sight tracking target position (for example, within 2 meters);

[0140] Based on the following indication direction vector, the smart car is controlled to track the sight line to follow the target, and the second laser radar data in the tracking process is obtained; wherein, based on the following indication direction vector, when the smart car is controlled to track the sight line to follow the target, the smart car is controlled to reach a vector starting point of a certain following indication direction vector based on the shortest distance, and after reaching the starting point, the direction of the smart car is kept consistent with the following indication direction vector;

[0141] Supplementing the mapping according to the second laser radar data and the indoor dynamic environment mapping model; wherein, when supplementing the mapping, the display area of ​​the display device is evenly divided into two left and right areas, the left side displays the mapping result, and the right side displays the supplemented mapping result;

[0142] The conditions for determining the target direction vector to follow the indication direction vector include:

[0143] The smart car can detect the sight-tracking target based on the vector starting position of the tracking indication direction vector; wherein, when there is no obstruction between the vector starting position of the tracking indication direction vector and the sight-tracking target position and the distance between them is within the farthest scanning distance of the laser radar, it is determined that the sight-tracking target can be detected;

[0144] and,

[0145] The position distance between the vector starting point position of the tracking indication direction vector and the sight tracking target position is greater than or equal to a preset position distance threshold; wherein the preset position distance threshold is manually set, such as: twice the radius length of the sight tracking target;

[0146] and,

[0147] When the position distance between the vector starting point position of the tracking indication direction vector and the sight tracking target position is greater than a preset position distance threshold, the vector angle between the tracking indication direction vector and the target direction vector is less than a preset vector angle threshold; wherein the preset vector angle threshold is manually preset, for example: 5°;

[0148] When the position distance between the vector starting point position following the indicated direction vector and the sight following target position is equal to the position distance threshold, the vector included angle range between the indicated direction vector and the target direction vector is linearly distributed within a preset range interval. Among them, the preset range interval linear distribution is, for example: an arithmetic sequence with a common difference of 10° between 0° and 180°.

[0149] The working principle and beneficial effects of the above technical solution are as follows:

[0150] When the target person's sight follows an environmental target, global modeling cannot display details, so local mapping can be performed to determine the environmental target in the mapping result, and extract the sight information of the target person staying in the mapping result, determine the sight following target with the sight staying duration greater than the sight staying duration threshold, construct a target direction vector with the sight following target position as the starting point and the speed direction as the vector direction, and at the same time, control the intelligent vehicle to approach the sight following target.

[0151] When the distance between the two is within the radar scanning range of the intelligent vehicle and the position distance between the vector starting point position of the indicated direction vector and the sight following target position is greater than the position distance threshold, the indicated direction vector is a vector with an included angle with the target direction vector less than the preset vector included angle threshold, that is, when the distance is far, the tracking can capture the sight following target;

[0152] When the position distance between the vector starting point position of the indicated direction vector and the sight following target position is equal to the position distance threshold, the vector included angle range between the indicated direction vector and the target direction vector is linearly distributed within a preset range interval, and the range interval linear distribution is an angle evenly linearly distributed between 0° and 180°, that is, when the distance is close, three-dimensional data of all aspects of the sight following target is captured.

[0153] The present invention determines the indicated direction vector of the target direction vector, and based on the indicated direction vector, controls the intelligent vehicle to track the sight following target, further improving the mapping accuracy and efficiency of supplementary mapping.

[0154] An embodiment of the present invention provides an indoor dynamic environment mapping system based on deep learning, as Figure 2 shown, including:

[0155] A mapping model construction subsystem 1, configured to construct an indoor dynamic environment mapping model based on deep learning technology;

[0156] A mapping requirement creation subsystem 2, configured to instruct the target person to create a mapping requirement;

[0157] A mapping result generation subsystem 3, configured to, after creation, input the mapping requirement into the indoor dynamic environment mapping model to generate a mapping result.

[0158] In one embodiment, the mapping model construction subsystem constructs an indoor dynamic environment mapping model based on deep learning technology and performs the following operations:

[0159] Lightweight improvement is made to the backbone network of YOLOv5 based on the PP-LCNet module to obtain the improved YOLOv5;

[0160] The improved YOLOv5 is trained and tested with the COCO dataset and the indoor dataset to select a lightweight network that meets the real-time detection conditions;

[0161] Based on the lightweight network, the frameworks of VSLAM and ORB-SLAM3 in a dynamic environment are designed to adapt to the dynamic environment, and at the same time, the overlapping of static and dynamic frames is used as the judgment mechanism;

[0162] Based on the judgment mechanism and the improved multi-view geometry dynamic region detection module, the indoor dynamic environment mapping model is determined.

[0163] In one embodiment, the mapping requirement creation subsystem creates mapping requirements and performs the following operations:

[0164] Obtain the tracking dialogue information of the mapping requirement space;

[0165] According to the tracking dialogue information, determine at least one mapping requirement semantics;

[0166] According to the mapping requirement semantics, instruct the target person to select the mapping area;

[0167] Plan the real-time cruise trajectory of the intelligent vehicle in the mapping area, trigger the intelligent vehicle to cruise based on the real-time cruise trajectory, and use the first lidar scan data of the intelligent vehicle during the cruise as the mapping requirement.

[0168] An indoor dynamic environment mapping system based on deep learning provided by an embodiment of the present invention also performs the following operations:

[0169] The supplementary mapping subsystem is used to parse the mapping result, determine the environmental target, and supplement the mapping according to the environmental target and the line-of-sight information of the target person.

[0170] In one embodiment, the supplementary mapping subsystem supplements the mapping according to the environmental target and the line-of-sight information of the target person, including:

[0171] According to the environmental target and the line-of-sight information of the target person, determine the line-of-sight following target of the target person;

[0172] Obtain the target direction vector of the line-of-sight following target;

[0173] Determine the following indication direction vector of the target direction vector;

[0174] Based on the following direction vector, control the intelligent vehicle to track the line-of-sight following target, and obtain the second lidar data during the tracking process;

[0175] Supplement the mapping according to the second lidar data and the indoor dynamic environment mapping model;

[0176] Among them, the determination conditions for the following direction vector for determining the target direction vector include:

[0177] The intelligent vehicle can detect the line-of-sight following target based on the vector starting point position of the following direction vector;

[0178] And,

[0179] The position distance between the vector starting point position of the following direction vector and the line-of-sight following target position is greater than or equal to the preset position distance threshold;

[0180] And,

[0181] When the position distance between the vector starting point position of the following direction vector and the line-of-sight following target position is greater than the preset position distance threshold, the vector angle between the following direction vector and the target direction vector is less than the preset vector angle threshold;

[0182] When the position distance between the vector starting point position of the following direction vector and the line-of-sight following target position is equal to the position distance threshold, the vector angle between the following direction vector and the target direction vector is linearly distributed within the preset range interval.

[0183] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention is also intended to include these changes and modifications.

Claims

1. A method for mapping indoor dynamic environments based on deep learning, characterized in that: include: Based on deep learning technology, build an indoor dynamic environment mapping model; Create map requirements; After the creation is completed, the mapping requirements are input into the indoor dynamic environment mapping model to generate the mapping results.

2. A method for mapping an indoor dynamic environment based on deep learning as claimed in claim 1, characterized in that: Based on deep learning technology, we build an indoor dynamic environment mapping model, including: Based on the PP-LCNet module, the backbone network of YOLOv5 is lightweight improved to obtain improved YOLOv5; The improved YOLOv5 is trained and tested using the COCO dataset and indoor dataset, and a lightweight network that meets the real-time detection conditions is selected; Design the framework of VSLAM and ORB-SLAM3 in dynamic environment based on lightweight network to adapt to dynamic environment, and use the overlap of dynamic and static frames as the judgment mechanism; Based on the judgment mechanism and the improved multi-view geometric dynamic area detection module, the indoor dynamic environment mapping model is determined.

3. The method for mapping an indoor dynamic environment based on deep learning according to claim 1, characterized in that: Create map requirements, including: Obtain tracking conversation information for the mapping requirement space; Determine at least one mapping requirement semantics based on the tracking conversation information; According to the semantics of mapping requirements, instruct the target personnel to select the mapping area; Plan the real-time cruising trajectory of the smart car in the mapping area, trigger the smart car to cruise based on the real-time cruising trajectory, and use the first lidar scanning data of the smart car during the cruising process as the mapping requirement.

4. The method for mapping an indoor dynamic environment based on deep learning according to claim 1, characterized in that: Also includes: Analyze the mapping results, determine the environmental targets, and supplement the mapping based on the environmental targets and the line of sight information of the target personnel.

5. The method for mapping an indoor dynamic environment based on deep learning as claimed in claim 4, characterized in that: Supplementary mapping based on environmental targets and target personnel’s line of sight information, including: According to the environmental target and the line of sight information of the target person, determine the line of sight tracking target of the target person; Get the target direction vector of the line of sight tracking target; Determine a following indication direction vector of the target direction vector; Based on the following indicated direction vector, the smart car is controlled to track the target and obtain the second laser radar data during the tracking process; Supplementary mapping based on the second laser radar data and the indoor dynamic environment mapping model; The conditions for determining the target direction vector to follow the indication direction vector include: The smart car can detect the sight-following target based on the vector starting position of the vector following the indicated direction; and, The position distance between the vector starting point position of the tracking indication direction vector and the sight tracking target position is greater than or equal to a preset position distance threshold; and, When the position distance between the vector starting point position of the tracking indication direction vector and the sight tracking target position is greater than a preset position distance threshold, the vector angle between the tracking indication direction vector and the target direction vector is less than a preset vector angle threshold; When the position distance between the vector starting point position of the tracking indication direction vector and the sight tracking target position is equal to the position distance threshold, the vector angle range between the tracking indication direction vector and the target direction vector is linearly distributed within a preset range interval.

6. A deep learning-based indoor dynamic environment mapping system, characterized in that: include: The mapping model building subsystem is used to build a mapping model of the indoor dynamic environment based on deep learning technology; The mapping requirement creation subsystem is used to instruct the target personnel to create mapping requirements; The mapping result generation subsystem is used to input the mapping requirements into the indoor dynamic environment mapping model after creation to generate the mapping results.

7. The indoor dynamic environment mapping system based on deep learning as claimed in claim 6, characterized in that: The mapping model building subsystem is based on deep learning technology and builds an indoor dynamic environment mapping model to perform the following operations: Based on the PP-LCNet module, the backbone network of YOLOv5 is lightweight improved to obtain improved YOLOv5; The improved YOLOv5 is trained and tested using the COCO dataset and indoor dataset, and a lightweight network that meets the real-time detection conditions is selected; Design the framework of VSLAM and ORB-SLAM3 in dynamic environment based on lightweight network to adapt to dynamic environment, and use the overlap of dynamic and static frames as the judgment mechanism; Based on the judgment mechanism and the improved multi-view geometric dynamic area detection module, the indoor dynamic environment mapping model is determined.

8. The indoor dynamic environment mapping system based on deep learning as claimed in claim 6, characterized in that: To create a mapping requirement in the subsystem, perform the following operations: Obtain tracking conversation information for the mapping requirement space; Determine at least one mapping requirement semantics based on the tracking conversation information; According to the semantics of mapping requirements, instruct the target personnel to select the mapping area; Plan the real-time cruising trajectory of the smart car in the mapping area, trigger the smart car to cruise based on the real-time cruising trajectory, and use the first lidar scanning data of the smart car during the cruising process as the mapping requirement.

9. The indoor dynamic environment mapping system based on deep learning as claimed in claim 6, characterized in that: Also perform the following operations: The supplementary mapping subsystem is used to analyze the mapping results, determine the environmental targets, and supplement the mapping based on the environmental targets and the line of sight information of the target personnel.

10. The indoor dynamic environment mapping system based on deep learning according to claim 9, characterized in that: The supplementary mapping subsystem supplements the mapping based on the line of sight information of the environment target and the target personnel, including: According to the environmental target and the line of sight information of the target person, determine the line of sight tracking target of the target person; Get the target direction vector of the line of sight tracking target; Determine a following indication direction vector of the target direction vector; Based on the following indicated direction vector, the smart car is controlled to track the target and obtain the second laser radar data during the tracking process; Supplementary mapping based on the second laser radar data and the indoor dynamic environment mapping model; The conditions for determining the target direction vector to follow the indication direction vector include: The smart car can detect the sight-following target based on the vector starting position of the vector following the indicated direction; and, The position distance between the vector starting point position of the tracking indication direction vector and the sight tracking target position is greater than or equal to a preset position distance threshold; and, When the position distance between the vector starting point position of the tracking indication direction vector and the sight tracking target position is greater than a preset position distance threshold, the vector angle between the tracking indication direction vector and the target direction vector is less than a preset vector angle threshold; When the position distance between the vector starting point position of the tracking indication direction vector and the sight tracking target position is equal to the position distance threshold, the vector angle range between the tracking indication direction vector and the target direction vector is linearly distributed within a preset range interval.

Images