Method and system for constructing complex scene space based on multi-sensor data fusion

Abstract

The application relates to a complex scene space construction method and system based on multi-sensor data fusion, and relates to the field of artificial intelligence, which comprises the following steps: collecting carrying information and a current position of a preset carrying device; based on the carrying information, the carrying object position and the carrying terminal are called; based on the current position of the device, the carrying object position, the carrying terminal and a preset carrying site terrain, a device carrying route is generated; the carrying device is controlled to travel along the device carrying route, and front image information is collected; only when the front image information contains a preset obstacle feature, a preset route planning method is used to update the device carrying route, and the carrying device is controlled to travel along the updated device carrying route. The application has the effect of improving the efficiency of luggage transportation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method and system for constructing complex scene spaces based on multi-sensor data fusion. Background Technology

[0002] Complex scene space construction refers to the process of collecting environmental data through sensors, combining algorithm processing and modeling techniques to construct a three-dimensional spatial model with geometric structure, semantic information, and dynamic features. It is widely used in fields such as autonomous driving and intelligent robots.

[0003] In airport baggage transportation, baggage needs to go through multiple stages such as security check, sorting, conveyor belt transfer, and loading. From security check, sorting, conveyor belt transfer to loading, a large amount of manual labor is required to transport the baggage to the correct location.

[0004] However, manual transport is prone to errors such as missorting or missing luggage due to visual fatigue or negligence, which reduces the efficiency of luggage transport and needs to be improved. Summary of the Invention

[0005] To improve the efficiency of baggage transportation, this invention provides a method and system for constructing complex scene spaces based on multi-sensor data fusion.

[0006] In a first aspect, the present invention provides a method for constructing complex scene spaces based on multi-sensor data fusion, employing the following technical solution:

[0007] A method for constructing complex scene spaces based on multi-sensor data fusion, comprising:

[0008] S1: Collects transport information and the current location of the preset transport device;

[0009] S2: Based on the transport information, retrieve the location and destination of the transported items;

[0010] S3: Generate a device transport route based on the current location of the device, the location of the transported items, the transport destination, and the preset terrain of the transport site;

[0011] S4: Control the conveying device to travel along the conveying route and collect image information in front of it;

[0012] S5: If and only if the image information in front contains preset obstacle features, update the device transport route using a preset route planning method, and control the transport device to travel along the updated device transport route.

[0013] By adopting the technical scheme, the system first collects the carrying information and the current position of the device, generates an initial carrying route in combination with the carrying site terrain, and provides clear travel guidance for the carrying device. In the device travel process, the front image information is continuously collected and the obstacle is identified. Once the obstacle is detected, the route planning method is started to update the path, so as to ensure that the carrying device can timely avoid the obstacle and continue to operate efficiently. In this way, real-time perception and dynamic response to complex scenes are realized, and the efficiency of luggage transportation is improved.

[0014] Optionally, the route planning method comprises:

[0015] S50: obtaining an obstacle type according to the front image information and the obstacle feature;

[0016] S51: when the obstacle type is a preset route center obstacle, collecting a left side free space and a right side free space;

[0017] S510: when the left side free space and the right side free space both exceed a preset required action gap, obtaining two side moving distances according to the front image information, the left side free space and the right side free space;

[0018] S5100: obtaining a rotation direction based on the two side moving distances;

[0019] S5101: updating the device carrying route based on the rotation direction, the two side moving distances, the carrying article location, the carrying terminal point and the carrying site terrain.

[0020] Optionally, it further comprises:

[0021] S511: when the left side free space and the right side free space both do not exceed the preset required action gap, collecting an obstacle height value;

[0022] S5110: when the obstacle height value is lower than a preset reference height value, determining whether the left side free space and the right side free space are greater than a preset device action gap;

[0023] S5111: when the left side free space and the right side free space are greater than the device action gap, obtaining two side moving distances according to the front image information, the left side free space and the right side free space, and performing S5100 to S5101, while performing article carrying by a preset lifting carrying method;

[0024] S5112: when the left side free space and the right side free space are not greater than the device action gap, reporting an obstacle abnormality prompt.

[0025] Optionally, the lifting carrying method comprises:

[0026] S51110: collecting item image information;

[0027] S51111: scanning and identifying preset item features from the item image information to obtain item dimensions;

[0028] S51112: generating a lifting height value based on the item dimensions and the obstacle height value;

[0029] S51113: controlling the carrying device to carry and lift at the lifting height value, and driving along an updated device carrying route after lifting.

[0030] Optionally, the method further comprises:

[0031] S52: when the obstacle type is a preset moving obstacle, collecting an obstacle moving speed and an obstacle dimension;

[0032] S520: obtaining an obstacle disappearance duration based on the obstacle moving speed and the obstacle dimension;

[0033] S521: collecting an obstacle distance;

[0034] S522: generating a device evading driving speed based on the obstacle distance, the obstacle disappearance duration, and a preset safety distance;

[0035] S523: controlling the carrying device to drive along the device carrying route at the device evading driving speed.

[0036] Optionally, the method further comprises:

[0037] S60: collecting driving image information;

[0038] S61: reporting a road surface steepness prompt when and only when the driving image information contains a preset steepness feature, and scanning and identifying the steepness feature from the driving image information to obtain a road surface steepness value;

[0039] S610: when the road surface steepness value does not exceed a preset reference steepness value, controlling the carrying device to continue driving along the device carrying route;

[0040] S611: when the road surface steepness value exceeds a preset reference steepness value, performing route planning by a preset steepness route planning method to obtain a steepness driving route;

[0041] S612: controlling the carrying device to drive along the steepness driving route.

[0042] Optionally, the steep route planning method comprises:

[0043] S6110: Collecting X-ray image information of the article;

[0044] S6111: Obtaining the article-in-box condition based on the X-ray image information of the article;

[0045] S6112: When the article-in-box condition contains the preset fragile article, determining the filler color, the filler contour, and the filler width based on the X-ray image information of the article and the article-in-box condition;

[0046] S6113: Obtaining the filler density based on the filler color;

[0047] S6114: Generating the anti-shock parameter based on the article-in-box condition, the filler contour, the filler density, and the filler width;

[0048] S6115: Obtaining the driving steep value based on the anti-shock parameter and the preset device driving speed;

[0049] S6116: When the road surface steep value does not exceed the driving steep value, defining the device carrying route as the steep driving route.

[0050] Optionally, the steep route planning method further comprises:

[0051] S6117: When the road surface steep value exceeds the driving steep value, updating the driving steep value based on the anti-shock parameter and the preset device minimum driving speed;

[0052] S61170: When the road surface steep value does not exceed the updated driving steep value, defining the device carrying route as the steep driving route;

[0053] S611700: Controlling the carrying device to drive along the steep driving route at the device minimum driving speed and collecting steep driving image information;

[0054] S611701: Controlling the carrying device to drive along the steep driving route at the device driving speed only when the steep driving image information does not contain the steep feature;

[0055] S61171: When the road surface steep value exceeds the updated driving steep value, reporting a steep abnormality prompt.

[0056] Optionally, it further comprises:

[0057] S70: Collecting the device driving position of the carrying device;

[0058] S71: collecting area image information of a preset luggage placement area on a preset centralized transport vehicle when the device running position coincides with the transport article site;

[0059] S72: collecting transport image information if and only if the area image information does not contain a preset transport device feature;

[0060] S73: scanning and identifying a preset slope feature from the transport image information to obtain a slope position;

[0061] S74: generating a climbing route based on the device running position, the slope position, and the luggage placement area, and controlling the transport device to go to the luggage placement area along the climbing route;

[0062] S75: when the transport device reaches the luggage placement area, controlling the transport device to clamp the articles in the luggage placement area to a preset transport device.

[0063] In a second aspect, the application provides a complex scene space construction system based on multi-sensor data fusion, which adopts the following technical solution:

[0064] A complex scene space construction system based on multi-sensor data fusion, comprising:

[0065] An acquisition module for acquiring transport information, current device position, and front image information;

[0066] A memory for storing programs for implementing any of the above complex scene space construction methods based on multi-sensor data fusion;

[0067] A processor for loading and executing the programs stored in the memory.

[0068] In summary, the application includes at least one of the following beneficial technical effects:

[0069] 1. By adopting the above technical solution, the system first acquires transport information and current device position, generates an initial transport route in combination with the transport site terrain, and provides clear running guidance for the transport device. During device running, the front image information is continuously acquired and obstacle identification is performed. Once an obstacle is detected, the route planning method is immediately started to update the path, ensuring that the transport device can timely avoid obstacles and continue to operate efficiently. In this way, real-time perception and dynamic response to complex scenes are achieved, thereby improving the efficiency of luggage transportation.

[0070] 2. The system first performs in-depth analysis of the image information in front of it, and accurately identifies the type of obstacle by combining obstacle features. When an obstacle in the center of the route is detected, the system simultaneously collects the available gaps on the left and right sides. By comparing these gaps with the required clearance, it determines whether detour is possible. If both sides meet the clearance requirements, the system further calculates the movement distance on both sides to determine the optimal turning direction. This allows the system to consider not only the obstacle position and movement distance when updating the transport route, but also global factors such as the location of the transported items, the transport endpoint, and the terrain of the transport site. This ensures that the new route meets both obstacle avoidance requirements and overall transport objectives, thus providing a reliable automated solution for scenarios such as airport baggage transport and further promoting the development of intelligent logistics.

[0071] 3. By first acquiring X-ray images of the items, deep learning algorithms are used to analyze the contents of the container and accurately identify fragile items. For fragile items, features such as the color, outline, and width of the filling material are further extracted. The filling material density is obtained by combining the color-density mapping relationship, thereby constructing a multi-dimensional seismic parameter model. By combining the seismic parameters with the device's travel speed, a personalized travel steepness value is generated as a key threshold for route selection. When the road steepness value does not exceed the travel steepness value, a steep travel route is selected, ensuring the safety of fragile items during transportation and avoiding efficiency losses due to excessive detours, thus improving the adaptability and reliability of the logistics system in complex scenarios. Attached Figure Description

[0072] Figure 1 This is a flowchart of a method for constructing a complex scene space based on multi-sensor data fusion in an embodiment of the present invention;

[0073] Figure 2 This is a simplified schematic diagram of the device transport route in an embodiment of the present invention;

[0074] Figure 3 This is a schematic diagram of a route in an embodiment of the present invention, where the obstacle type is a route center obstacle. Detailed Implementation

[0075] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0076] Reference Figure 1 and Figure 2 This application discloses a method for constructing a complex scene space based on multi-sensor data fusion, including the following steps:

[0077] S1: Collects transport information and the current location of the preset transport device.

[0078] The carrying information refers to the information related to the carrying task of the carrying device. The carrying information can be obtained from the preset carrying terminal. The carrying terminal stores the carrying information. When a carrying task occurs, the information related to the task is recorded in the carrying terminal for retrieval. The carrying terminal refers to the terminal for storing the carrying information. The carrying terminal is preset by the person skilled in the art, which is not described herein. In this embodiment, the carrying object is a luggage.

[0079] The carrying device refers to a mobile robot for carrying objects. The current position of the device refers to the current position of the carrying device. The current position of the device is obtained by the preset GPS positioning chip on the carrying device.

[0080] S2: based on the carrying information, the carrying object location and the carrying terminal are obtained.

[0081] The carrying object location refers to the location of the object to be carried by the carrying device. The carrying terminal refers to the position where the carrying device carries the object to be carried. The carrying object location and the carrying terminal can be obtained by understanding the carrying information, which contains the carrying object location and the carrying terminal.

[0082] S3: based on the current position of the device, the carrying object location, the carrying terminal and the preset carrying site topography, the device carrying route is generated.

[0083] The carrying site topography refers to the topographic features and layout information of the physical space environment where the carrying device performs the carrying task. The carrying site topography is preset by the person skilled in the art, which is not described herein. The device carrying route refers to the driving route of the carrying device when performing the carrying task. The device carrying route can be generated by inputting the current position of the device, the carrying object location, the carrying terminal and the carrying site topography into the preset path planning algorithm. In this embodiment, the path planning algorithm can be A* algorithm. The A* algorithm is well known in the art, which is not described herein.

[0084] S4: control the carrying device to drive along the device carrying route, and collect the front image information.

[0085] The front image information refers to the image presented in front of the carrying device during driving. The front image information is obtained by the camera on the carrying device.

[0086] When the carrying device is controlled to drive along the device carrying route, the front image information needs to be collected at the same time for subsequent steps.

[0087] S5: when and only when the preset obstacle features are contained in the front image information, the device carrying route is updated by the preset route planning method, and the carrying device is controlled to drive along the updated device carrying route.

[0088] Obstacle feature refers to a visual data feature used to identify an obstacle in the travel path of the carrying device. The obstacle feature is set by a person skilled in the art in advance, which is not described herein. Route planning method refers to a method for re-planning the route of the carrying device when the obstacle feature is contained in the front image information. The specific route planning method is described in detail in subsequent S50 to S5112 and S52 to S523, which is not described herein.

[0089] When and only when the obstacle feature is contained in the front image information, it indicates that there is an obstacle blocking the carrying device on the road along the device carrying route, and the device carrying route needs to be updated by the route planning method, and the carrying device is controlled to travel along the updated device carrying route.

[0090] Reference Figure 3 The route planning method comprises the following steps:

[0091] S50: obtaining an obstacle type according to the front image information and the obstacle feature.

[0092] Obstacle type refers to the specific type of obstacle classified according to different features and attributes. In this embodiment, the obstacle type includes route center obstacle and moving obstacle. The route center obstacle refers to an obstacle located in the center of the device carrying route and stationary. The moving obstacle refers to an obstacle falling into the front image information and moving.

[0093] Since the front image information is continuously collected, the position change of the obstacle in two adjacent images can be understood to know whether the obstacle is a moving obstacle. When the position of the obstacle in two adjacent images changes, it indicates that it is a moving obstacle. It is a common knowledge in the art to determine whether the obstacle is a moving obstacle through continuous images, which is not described herein.

[0094] By understanding the position of the obstacle feature in the front image information and whether it moves, the obstacle type can be known.

[0095] S51: When the obstacle type is a preset route center obstacle, collect the left side clearance and the right side clearance.

[0096] The left side clearance refers to the space gap in the left side of the route center line that is not occupied by the obstacle and can be used for the device to pass when the obstacle is located in the center of the current planned route of the carrying device.

[0097] The right side clearance refers to the space gap in the right side of the route center line that is not occupied by the obstacle and can be used for the device to pass when the obstacle is located in the center of the current planned route of the carrying device.

[0098] The left empty gap and the right empty gap are scanned by a preset infrared scanner on the carrying device. The infrared scanning technology is well known in the art and will not be described here.

[0099] When the obstacle type is the route center obstacle, it indicates that the center of the device carrying route is blocked by the obstacle, and the left empty gap and the right empty gap need to be collected for subsequent steps.

[0100] S510: When the left empty gap and the right empty gap are both beyond the preset required action gap, the two-side moving distances are obtained according to the front image information, the left empty gap and the right empty gap.

[0101] The required action gap refers to the minimum width threshold requirement of the passable space on both sides of the carrying device when encountering a route center obstacle in order to safely and smoothly complete the detour operation. The required action gap is set by a person skilled in the art in advance and will not be described here. The two-side moving distances refer to the moving distances of the carrying device when performing the detour operation to the left empty gap and the right empty gap, respectively. By understanding the front image information and the left empty gap and the right empty gap in the image, the distance between the carrying device and the left empty gap and the right empty gap in the front image information can be known, and by understanding the shooting scale of the front image information, the actual distance can be compared to obtain the two-side moving distances. The shooting scale of the front image information is set by a person skilled in the art in advance and will not be described here.

[0102] When the left empty gap and the right empty gap are both beyond the required action gap, it indicates that the carrying device can choose to detour to the left and right sides, and the two-side moving distances need to be obtained first for subsequent steps.

[0103] S5100: Obtain the turning direction based on the two-side moving distances.

[0104] The turning direction refers to the direction of the carrying device when performing the detour. The turning direction can be obtained by understanding the size relationship between the two-side moving distances. When the distance of the left empty gap is less than the distance of the right empty gap, the left empty gap is selected for detour, and the turning direction is left. Conversely, the turning direction is right. When the moving distances of the left and right sides are the same, the left side is preferred.

[0105] S5101: Update the device carrying route based on the turning direction, the two-side moving distances, the carrying article location, the carrying end point and the carrying site topography.

[0106] A new device carrying route can be generated by inputting the rotation direction, the moving distance on both sides, the carrying object location, the carrying end point, and the carrying site topography into the path planning algorithm, so as to update the device carrying route.

[0107] The method further comprises the following steps:

[0108] S511: When the left free gap and the right free gap do not exceed the preset required action gap, the obstacle height value is collected.

[0109] The obstacle height value refers to the height of the obstacle. The obstacle height value is obtained by scanning with an infrared scanner on the carrying device.

[0110] When the left free gap and the right free gap do not exceed the required action gap, it indicates that the carrying device cannot directly make a detour on both sides, and the obstacle height value needs to be collected first for subsequent steps.

[0111] When one of the left free gap and the right free gap exceeds the required action gap, a detour can be directly made in the direction exceeding the required action gap.

[0112] S5110: When the obstacle height value is lower than a preset reference height value, it is determined whether the left free gap and the right free gap are greater than a preset device action gap.

[0113] The reference height value refers to a reference threshold for detecting whether the obstacle is too high. The device action gap refers to the size of the gap that the carrying device can pass through. The reference height value and the device action gap are both set by a person skilled in the art in advance and will not be described here.

[0114] When the obstacle height value is lower than the reference height value, it indicates that the obstacle is not too high, and it is necessary to determine whether the left free gap and the right free gap are greater than the device action gap.

[0115] S5111: When the left free gap and the right free gap are both greater than the device action gap, the moving distance on both sides is obtained according to the image information in front, the left free gap, and the right free gap, and S5100 to S5101 are executed, and the object is carried by a preset lifting carrying method.

[0116] The lifting carrying method refers to a method of lifting the carried object to pass through the gap and then carrying. The specific lifting carrying method is described in detail in subsequent S51110 to S51113 and will not be described here.

[0117] When both the left empty gap and the right empty gap are greater than the device action gap, it indicates that the sizes of the two empty gaps can be used for the passing of the carrying device, and the two side moving distances are obtained first, and S5100 to S5101 are executed to obtain a new device carrying route, and the carrying device carries the articles by the lifting carrying method.

[0118] The method for obtaining the two side moving distances in this step is the same as S510 described above, and will not be described here.

[0119] S5112: When both the left empty gap and the right empty gap are not greater than the device action gap, an obstacle abnormality prompt is reported.

[0120] When both the left empty gap and the right empty gap are not greater than the device action gap, it indicates that the sizes of the two empty gaps cannot be used for the passing of the carrying device, and an obstacle abnormality prompt needs to be reported.

[0121] When one of the left empty gap and the right empty gap is greater than the device action gap, the direction exceeding the device action gap can be directly selected for detouring.

[0122] The lifting carrying method includes the following steps:

[0123] S51110: Collecting article image information.

[0124] The article image information refers to the image of the article carried by the carrying device. The article image information is obtained by shooting through the camera on the carrying device.

[0125] S51111: Scanning and identifying the preset article features from the article image information to obtain the article size.

[0126] The article features refer to the shape features of the carried article. The article features are set by the person skilled in the art in advance, and will not be described here. The article size refers to the size of the carried article. The article size can be obtained by scanning and identifying the article features from the article image information. The image recognition technology is well known in the art, and will not be described here.

[0127] S51112: Generating a lifting height value based on the article size and the obstacle height value.

[0128] The lifting height value refers to the height value at which the carrying device needs to lift the article. By understanding the article size, the size in the vertical direction of the article can be known, and the size in the vertical direction plus the obstacle height value can obtain the lifting height value.

[0129] S51113: Controlling the carrying device to carry and lift at the lifting height value, and driving along the updated device carrying route after lifting.

[0130] The carrying device is controlled to carry out the carrying lifting at the lifting height value, and after the lifting, the carrying device is controlled to travel along the updated device carrying route.

[0131] The method further comprises the following steps:

[0132] S52: When the obstacle type is the preset moving obstacle, the obstacle moving speed and the obstacle size are collected.

[0133] The obstacle moving speed refers to the moving speed of the moving obstacle. The obstacle size refers to the size of the moving obstacle. The obstacle moving speed is measured by the laser radar on the carrying device. The obstacle size is measured by the infrared scanner scanning on the carrying device.

[0134] When the obstacle type is the moving obstacle, it means that the obstacle is moving, and the obstacle moving speed and the obstacle size need to be collected for subsequent steps.

[0135] S520: Based on the obstacle moving speed and the obstacle size, the obstacle disappearance duration is obtained.

[0136] The obstacle disappearance duration refers to the time required for the moving obstacle to completely leave the preset detection range of the carrying device from the current time. The detection range refers to the spatial range in which the carrying device can realize real-time perception and response to the obstacle through the laser radar. The detection range is set by the person skilled in the art in advance, and is not described here.

[0137] By knowing the obstacle size and the detection range, the distance between the obstacle and the boundary of the detection range can be known. Then, the distance is divided by the obstacle moving speed to obtain the obstacle disappearance duration.

[0138] S521: The obstacle distance is collected.

[0139] The obstacle distance refers to the distance between the carrying device and the moving obstacle. The obstacle distance can be measured by the laser radar.

[0140] S522: Based on the obstacle distance, the obstacle disappearance duration, and the preset safety distance, the device evading travel speed is generated.

[0141] The safety distance refers to the distance that needs to be kept between the carrying device and the moving obstacle. The safety distance is set by the person skilled in the art in advance, and is not described here. The device evading travel speed refers to the reasonable travel speed calculated to ensure the safety distance with the obstacle and to pass safely within the obstacle disappearance duration when the carrying device detects the moving obstacle. The device evading travel speed can be obtained by calculating the difference between the obstacle distance and the safety distance, and then dividing the difference by the obstacle disappearance duration.

[0142] S523: control the carrying device to travel along the device carrying route at the device evasive travel speed.

[0143] The carrying device is controlled to travel along the device carrying route at the device evasive travel speed to evade the moving obstacle, and after the obstacle evasion is completed, the carrying device is controlled to return to the preset normal travel speed. The normal travel speed refers to the travel speed of the carrying device without any adjustment. The normal travel speed is set by the person skilled in the art in advance, and is not described herein.

[0144] Further comprising the following steps:

[0145] S60: collect travel image information.

[0146] The travel image information refers to the image of the carrying device during travel. The travel image information is obtained by shooting by a camera.

[0147] S61: report the road steepness prompt only when the travel image information contains the preset steepness feature, and scan and identify the steepness feature from the travel image information to obtain the road steepness value.

[0148] The steepness feature refers to the visual feature when the road is steep. The steepness feature is set by the person skilled in the art in advance, and is not described herein. The road steepness value refers to a quantitative value for indicating the degree of road steepness. By geometrically measuring the steepness feature from the travel image information and combining the preset calibration parameters of the camera, the actual physical value can be converted, and then the road steepness value can be obtained. The preset calibration parameters of the camera are set by the person skilled in the art in advance, and are not described herein. The geometric measurement based on the image is well known in the art, and is not described herein.

[0149] Only when the travel image information contains the steepness feature, it means that the road is steep, the road steepness prompt needs to be reported, and the steepness feature is scanned and identified from the travel image information to obtain the road steepness value.

[0150] S610: control the carrying device to continue traveling along the device carrying route when the road steepness value does not exceed the preset reference steepness value.

[0151] The reference steepness value refers to a reference threshold for comparing whether the road is too steep. The reference steepness value is set by the person skilled in the art in advance, and is not described herein.

[0152] When the road steepness value does not exceed the reference steepness value, it means that the road is not too steep, and the carrying device can be controlled to continue traveling along the device carrying route.

[0153] S611: When the road steepness value exceeds the preset benchmark steepness value, route planning is performed using the preset steep route planning method to obtain a steep driving route.

[0154] A steep travel route refers to a route that is replanned when the road surface is too steep during the transport of the device. Steep travel routes can be obtained through steep route planning methods. Specific steep route planning methods will be explained in detail in subsequent sections S6110 to S611711, and will not be elaborated upon here.

[0155] When the road steepness value exceeds the benchmark steepness value, it indicates that the road surface is too steep, and route planning should be carried out using steep route planning methods to obtain a steep driving route.

[0156] S612: Control the transport device to travel along a steep route.

[0157] The transport device is controlled to travel along a steep route to complete the transport of objects.

[0158] Steep route planning methods include the following steps:

[0159] S6110: Collect X-ray image information of objects.

[0160] The X-ray image information of the item refers to the X-ray image obtained after scanning the item carried by the handling device using an X-ray imaging device (in this embodiment, the X-ray imaging device is a security inspection machine). The X-ray image information of the item can be obtained by scanning the item with an X-ray imaging device.

[0161] S6111: Obtain information about the contents of the container based on X-ray image information of the items.

[0162] The contents of a suitcase refer to the physical properties and spatial distribution of the items inside the packaging box or suitcase. This can be determined by extracting and analyzing features from X-ray images of the items. Feature extraction and analysis of X-ray images is common knowledge in this field and will not be elaborated upon here.

[0163] S6112: When the contents of the box include preset fragile items, determine the color, outline, and width of the filler based on the X-ray image information of the items and the contents of the box.

[0164] The fragile article refers to the physical attribute of the article in the luggage as a fragile article. The filling color refers to the color attribute of the material used to buffer the fragile article in the luggage (the clothes in the luggage can be considered as the filling). The color of each article in the article X-ray image information can be identified by the multispectral X-ray imaging technology, and then the filling color can be obtained. The filling contour refers to the contour of the material used to buffer the fragile article in the luggage. The fragile article can be segmented by the image segmentation technology, so as to obtain the filling contour. The filling width refers to the width of the material used to buffer the fragile article in the luggage. The filling width can be obtained by measuring the width of the filling from the article X-ray image information.

[0165] The multispectral X-ray imaging technology, the image segmentation technology, and the width measurement from the image are well known in the art, and will not be repeated here.

[0166] When the article in the luggage contains a fragile article, it indicates that there is a fragile article in the luggage to be transported, and the filling color, the filling contour, and the filling width need to be obtained first for subsequent steps.

[0167] S6113: obtaining the filling density based on the filling color.

[0168] The filling density refers to the mass of the filling material per unit volume. The filling density corresponding to the filling color can be queried by a pre-set density reference table. The table records different filling densities corresponding to different filling colors. The density reference table is formed by sequentially recording the different filling densities corresponding to different filling colors by those skilled in the art, and will not be repeated here.

[0169] S6114: generating the shock resistance parameter based on the article in the luggage, the filling contour, the filling density, and the filling width.

[0170] The shock resistance parameter refers to the quantitative value of the shock resistance protection capability of the filling in the luggage to the fragile article. By understanding the article in the luggage and the filling contour, the volume of the fragile article and the filling wheel can be known. The shock resistance parameter corresponding to the volume of the fragile article, the volume of the filling wheel, the filling density, and the filling width can be queried from a pre-set shock resistance reference table. The table records the shock resistance parameters corresponding to different volumes of fragile articles, different volumes of filling wheels, different filling densities, and different filling widths. The shock resistance reference table is formed by sequentially recording the shock resistance parameters corresponding to different volumes of fragile articles, different volumes of filling wheels, different filling densities, and different filling widths by those skilled in the art after testing, and will not be repeated here.

[0171] S6115: obtaining a running steepness value based on the anti-shock parameter and the preset device running speed.

[0172] The device running speed refers to the speed at which the carrying device runs. The device running speed is set by a person skilled in the art in advance, and will not be described here. The running steepness value refers to the maximum road steepness degree allowed for running of the carrying device when carrying fragile goods in combination with the anti-shock parameter of the packaging box and the device running speed.

[0173] The running steepness value corresponding to the anti-shock parameter and the device running speed can be queried through a preset running steepness comparison table. The comparison table records different running steepness values corresponding to different anti-shock parameters and device running speeds. The running steepness comparison table is formed by a person skilled in the art recording different running steepness values corresponding to different anti-shock parameters and device running speeds in turn after testing, and will not be described here.

[0174] S6116: defining the device carrying route as a steep running route when the road steepness value does not exceed the running steepness value.

[0175] When the road steepness value does not exceed the running steepness value, it means that the carrying device continues to run along the device carrying route at the device running speed, and will not affect the fragile goods in the box. Therefore, the device carrying route can be directly defined as a steep running route.

[0176] The steep route planning method further includes the following steps:

[0177] S6117: updating the running steepness value based on the anti-shock parameter and the preset device minimum running speed when the road steepness value exceeds the running steepness value.

[0178] The device minimum running speed refers to the minimum speed at which the carrying device needs to move. The device minimum running speed is set by a person skilled in the art in advance, and will not be described here. As described above in S6115, the running steepness value corresponding to the anti-shock parameter and the device minimum running speed can be deduced through the running steepness comparison table, so as to update the running steepness value.

[0179] When the road steepness value exceeds the running steepness value, it means that the carrying device continues to run along the device carrying route at the device running speed, which will affect the fragile goods in the box. Therefore, the running steepness value needs to be updated for subsequent steps.

[0180] S61170: defining the device carrying route as a steep running route when the road steepness value does not exceed the updated running steepness value.

[0181] When the road steepness value does not exceed the updated driving steepness value, it indicates that the carrying device continues to travel along the device carrying route at the device minimum driving speed, which does not affect the fragile items in the box, and the device carrying route is directly defined as the steep driving route.

[0182] S611700: Control the carrying device to travel along the steep driving route at the device minimum driving speed, and collect steep driving image information.

[0183] The steep driving image information refers to the image of the carrying device traveling on the steep road. The steep driving image information is obtained by shooting with a camera.

[0184] Control the carrying device to travel along the steep driving route at the device minimum driving speed, while continuously collecting steep driving image information for subsequent steps.

[0185] S611701: Control the carrying device to travel along the steep driving route at the device driving speed only when the steep driving image information does not contain steep features.

[0186] When the steep driving image information does not contain steep features, it indicates that the carrying device has left the steep road section, and the carrying device can be controlled to travel along the steep driving route at the device driving speed.

[0187] S61171: Report a steepness abnormality prompt when the road steepness value exceeds the updated driving steepness value.

[0188] When the road steepness value exceeds the updated driving steepness value, it indicates that the carrying device will affect the fragile items in the box even if it continues to travel along the device carrying route at the device minimum driving speed, and a steepness abnormality prompt needs to be reported.

[0189] Further comprising the following steps:

[0190] S70: Collect the device driving position of the carrying device.

[0191] The device driving position refers to the position where the carrying device is currently located. The device driving position is collected by the GPS positioning chip in the carrying device.

[0192] S71: When the device driving position is consistent with the carrying article location, collect the preset area image information of the preset luggage placement area on the centralized carrying vehicle.

[0193] The centralized carrying vehicle refers to a vehicle used to centrally place luggage boxes that need to be carried. The luggage placement area refers to the area on the centralized carrying vehicle for placing luggage. The luggage placement area is set by those skilled in the art in advance and is not described here. The area image information refers to the image of the luggage placement area. The area image information is obtained by shooting with a camera.

[0194] When the device driving position coincides with the luggage placement area, it indicates that the carrying device has reached the luggage placement area, and the area image information needs to be collected first.

[0195] S72: Collect the carrying image information only when the preset carrying device feature is not included in the area image information.

[0196] The carrying device feature refers to the contour feature of the carrying device. The carrying device feature is set by a person skilled in the art in advance, which is not described here. The carrying image information refers to the image containing the centralized carrying vehicle and the ramp feature for the carrying device to enter the centralized carrying vehicle. The carrying image information is obtained by shooting with a camera. The ramp feature refers to the contour feature of the ramp for the carrying device to enter the centralized carrying vehicle. The ramp feature is set by a person skilled in the art in advance, which is not described here.

[0197] When the carrying device feature is not included in the area image information, it indicates that there is no carrying device on the centralized carrying vehicle for luggage carrying, and the carrying image information needs to be collected first for the subsequent steps.

[0198] S73: Scan and identify the preset ramp feature from the carrying image information to obtain the ramp position.

[0199] The ramp position refers to the position of the ramp. The ramp feature is scanned and identified from the carrying image information and is marked to obtain the ramp position. Image recognition technology is well known in the art, which is not described here.

[0200] S74: Generate a ramp climbing route based on the device driving position, the ramp position, and the luggage placement area, and control the carrying device to go to the luggage placement area along the ramp climbing route.

[0201] The ramp climbing route refers to the route of the carrying device climbing the centralized carrying vehicle along the ramp. The device driving position, the ramp position, and the luggage placement area are input into the path planning algorithm to generate the ramp climbing route.

[0202] After the ramp climbing route is generated, the carrying device is controlled to go to the luggage placement area along the ramp climbing route for the subsequent steps.

[0203] S75: After the carrying device reaches the luggage placement area, control the carrying device to clamp the objects in the luggage placement area onto the preset transport device.

[0204] The transport device refers to a device for transporting luggage on the vehicle to the vehicle. When the carrying device reaches the luggage placement area, the carrying device needs to be controlled to clamp the objects in the luggage placement area onto the transport device, so as to transport the luggage in the luggage placement area to the vehicle.

[0205] Based on the same inventive concept, the embodiment of the present application provides a complex scene space construction system based on multi-sensor data fusion, comprising:

[0206] The acquisition module is used for acquiring the carrying information, the current position of the device, the image information in front, the left side gap, the right side gap, the obstacle height value, the article image information, the obstacle moving speed, the obstacle size, the obstacle distance, the driving image information, the article X-ray image information, the steep driving image information, the device driving position, the area image information and the carrying image information.

[0207] The memory is used for storing the program for realizing the complex scene space construction method based on multi-sensor data fusion.

[0208] The processor is used for loading and executing the program stored in the memory.

[0209] Those skilled in the art can clearly understand that, for the convenience and brevity, only the above-mentioned division of each functional module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional modules according to the needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0210] The above is only the preferred embodiment of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any technical solution falling within the concept of the present application shall be considered as falling within the protection scope of the present application. It should be noted that, for ordinary skilled in the art, some improvements and decorations without departing from the principle of the present application shall also be considered as falling within the protection scope of the present application.

Claims

1. A method for constructing a complex scene space based on multi-sensor data fusion, characterized in that, include: S1: Collects transport information and the current location of the preset transport device; S2: Based on the transport information, retrieve the location and destination of the transported items; S3: Generate a device transport route based on the current location of the device, the location of the transported items, the transport destination, and the preset terrain of the transport site; S4: Control the conveying device to travel along the conveying route and collect image information in front of it; S5: If and only if the image information in front contains preset obstacle features, update the device transport route using a preset route planning method, and control the transport device to travel along the updated device transport route. Also includes: S60: Acquires driving image information; S61: If and only if the driving image information contains a preset steep feature, report a steep road surface warning and scan and identify the steep feature from the driving image information to obtain a steep road surface value; S610: When the road steepness value does not exceed the preset benchmark steepness value, control the conveying device to continue traveling along the device conveying route; S611: When the road steepness value exceeds the preset benchmark steepness value, route planning is performed using a preset steep route planning method to obtain a steep driving route. S612: Control the transport device to travel along the steep travel route; The steep route planning method includes: S6110: Acquire X-ray image information of objects; S6111: Obtain the contents of the box based on the X-ray image information of the item; S6112: When the contents of the box include preset fragile items, the color, outline, and width of the filler are determined based on the X-ray image information of the items and the contents of the box. S6113: Obtain the filler density based on the filler color; S6114: Based on the contents of the box, the outline of the filler, the density of the filler, and the width of the filler, generate shock resistance parameters. The shock resistance parameters refer to the quantitative values ​​of the shock resistance protection capability of the filler inside the box or suitcase for fragile items. S6115: Based on the seismic parameters and the preset device travel speed, the travel steepness value is obtained. The travel steepness value refers to the quantified value of the maximum road surface steepness that the transport device can travel on when carrying fragile items, combined with the seismic parameters of the packaging box and luggage and the device travel speed. The travel steepness value corresponding to the seismic parameters and device travel speed can be found through the preset travel steepness comparison table. The comparison table records different travel steepness values ​​corresponding to different seismic parameters and device travel speeds. The travel steepness comparison table is a standardized data table formed by systematically testing different travel steepness values ​​corresponding to different seismic parameters and device travel speeds. The table clearly records the unique travel steepness value corresponding to different combinations of seismic parameters and device travel speeds. S6116: When the road steepness value does not exceed the driving steepness value, the device transport route is defined as a steep driving route.

2. The method for constructing complex scene space based on multi-sensor data fusion according to claim 1, characterized in that, The route planning method includes: S50: Obtain the obstacle type based on the image information in front and the obstacle features; S51: When the obstacle type is a preset route center obstacle, collect the left and right clearances; S510: When both the left and right clearances exceed the preset required movement gaps, the movement distance on both sides is obtained based on the frontal image information, the left clearance, and the right clearance. S5100: The rotation direction is obtained based on the moving distance on both sides; S5101: Update the device's transport route based on the rotation direction, the distance moved to both sides, the location of the transported item, the transport endpoint, and the terrain of the transport site.

3. The method for constructing complex scene space based on multi-sensor data fusion according to claim 2, characterized in that, Also includes: S511: When neither the left-side clearance nor the right-side clearance exceeds the preset required movement clearance, collect the obstacle height value; S5110: When the height of the obstacle is lower than the preset reference height, determine whether the left clearance and the right clearance are greater than the preset device movement clearance. S5111: When the left clearance and the right clearance are greater than the movement clearance of the device, the movement distance on both sides is obtained based on the front image information, the left clearance and the right clearance, and S5100 to S5101 are executed, while the item is moved using a preset lifting and moving method. S5112: When the left and right clearances are not greater than the device movement clearance, an obstacle abnormality warning is reported.

4. The method for constructing complex scene space based on multi-sensor data fusion according to claim 3, characterized in that, The lifting and transporting method includes: S51110: Collects image information of items; S51111: Scan and identify preset item features from the item image information to obtain the item size; S51112: Generate a lift height value based on the item size and the obstacle height value; S51113: Control the transport device to transport and lift at the lifting height value, and after lifting, travel along the updated transport route of the device.

5. The method for constructing complex scene space based on multi-sensor data fusion according to claim 2, characterized in that, Also includes: S52: When the obstacle type is a preset moving obstacle, collect the obstacle's moving speed and obstacle size; S520: Obtain the obstacle disappearance time based on the obstacle's moving speed and the obstacle's size. The obstacle disappearance time refers to the time required for the moving obstacle to completely leave the preset detection range of the conveying device from the current moment. S521: Collect distance to obstacles; S522: Generate an obstacle avoidance speed based on the obstacle distance, the obstacle disappearance time, and a preset safety distance; S523: Control the transport device to travel along the transport route at the device avoidance speed.

6. The method for constructing complex scene space based on multi-sensor data fusion according to claim 1, characterized in that, The steep route planning method also includes: S6117: When the road steepness value exceeds the driving steepness value, the driving steepness value is updated based on the seismic parameters and the preset minimum driving speed of the device. S61170: When the road steepness value does not exceed the updated driving steepness value, the device transport route is defined as a steep driving route; S611700: Control the transport device to travel along the steep travel route at the device's minimum travel speed, and collect steep travel image information; S611701: Control the transport device to travel along the steep driving route at the device driving speed if and only if the steep driving image information does not contain the steep feature; S61171: When the road steepness value exceeds the updated driving steepness value, a steepness anomaly warning is reported.

7. The method for constructing complex scene space based on multi-sensor data fusion according to claim 1, characterized in that, Also includes: S70: Collect the device travel position of the conveying device; S71: When the device travels to the same location as the transported items, it collects area image information of the preset luggage placement area on the preset centralized transport vehicle. S72: Acquire transport image information if and only if the area image information does not contain preset transport device features. Transport image information refers to an image containing features of a centralized transport vehicle and a ramp for the transport device to enter the centralized transport vehicle. S73: Scan and identify preset slope features from the transport image information to obtain the slope location; S74: Generate a climbing route based on the device's driving position, the slope position, and the luggage placement area, and control the transport device to travel along the climbing route to the luggage placement area; S75: When the handling device arrives at the luggage placement area, the handling device is controlled to clamp the items in the luggage placement area onto a preset transport device. The transport device is a device used to transport luggage from the vehicle to the vehicle.

8. A complex scene spatial construction system based on multi-sensor data fusion, characterized in that, include: The data acquisition module is used to collect information on the handling process, the current location of the device, and the image information in front of it. A memory for storing a program that implements the complex scene space construction method based on multi-sensor data fusion as described in any one of claims 1 to 7; The processor is used to load and execute programs stored in memory.

Images