Inventory map generation method and device, electronic equipment and storage medium

Abstract

The invention provides an inventory map generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining point cloud data in a target warehouse, motion data of the robot and image data in the target warehouse through a plurality of sensors carried by the robot in the moving process of the robot in the target warehouse; coupling the point cloud data and the motion data to obtain a geometric point cloud map in the target warehouse and motion track data of the robot; carrying out image recognition on the image data, and determining goods semantic information contained in the image; for each piece of goods semantic information, determining target image data corresponding to the goods semantic information, and performing multi-modal data fusion on the target image data, the geometric point cloud map and the motion track data to generate a semantic point cloud map of the target warehouse; and performing feature analysis on the semantic point cloud map to generate an inventory map of the target warehouse. According to the method and the device, the high-precision inventory map carrying semantic information in the warehouse is automatically constructed by fusing various sensor data, and core data support is provided for realizing refined and intelligent inventory management.

Description

Technical Field

[0001] This application relates to the field of inventory management technology, and in particular to a method, apparatus, electronic device and storage medium for generating an inventory map. Background Technology

[0002] With the rapid development of smart logistics and intelligent manufacturing, the digitalization of inventory management is the core of smart logistics and has become a key link in improving warehousing efficiency and reducing operating costs.

[0003] However, existing inventory information collection solutions for digital management mostly rely on single sensor modalities, leading to significant limitations in inventory management. For example, purely vision-based methods capture images of shelves with cameras, but their performance is heavily dependent on ambient lighting conditions, with recognition rates dropping significantly in dim, reflective, or low-texture scenes. Furthermore, this method struggles to obtain precise depth information and 3D dimensions of objects and exhibits poor robustness when faced with stacked or obstructed goods. Another example is LiDAR-based methods, which construct 3D point cloud models of the environment by emitting laser beams and measuring echo times. However, LiDAR cannot recognize semantic information; it only knows "there is something here," but not "what this thing is." Finally, RFID technology allows for contactless batch reading of tag information, but its spatial positioning accuracy is low, failing to provide information on item orientation, size, and specific placement, making it difficult to meet the needs of refined inventory management.

[0004] Therefore, current technologies still have gaps in areas such as how to generate high-precision and information-rich inventory maps specifically designed for inventory management scenarios. Summary of the Invention

[0005] In view of this, the purpose of this application is to provide a method, apparatus, electronic device and storage medium for generating an inventory map. By integrating data from multiple sensors, a high-precision inventory map with semantic information is automatically constructed within the warehouse, which can characterize the inventory structure and inventory status of the warehouse, and provide core data support for achieving refined and intelligent inventory management.

[0006] This application provides a method for generating an inventory map, the method comprising: During the robot's movement in the target warehouse, point cloud data, robot motion data, and image data of the target warehouse are acquired through various sensors mounted on the robot. By coupling the point cloud data and the motion data, a geometric point cloud map of the target warehouse and the robot's motion trajectory data are obtained. Image recognition is performed on the image data to determine the semantic information of the goods contained in the image; For each cargo semantic information, the target image data corresponding to the cargo semantic information is determined, and the target image data, the geometric point cloud map and the motion trajectory data are fused in a multimodal manner to generate the semantic point cloud map of the target warehouse; Feature analysis is performed on the semantic point cloud map to generate an inventory map of the target warehouse; wherein the inventory map represents the inventory structure and inventory status of the target warehouse.

[0007] This application embodiment also provides an apparatus for generating an inventory map, the apparatus comprising: The acquisition module is used to acquire point cloud data, trajectory data of the robot, and image data of the target warehouse through various sensors carried by the robot during the robot's movement in the target warehouse. A coupling module is used to couple the point cloud data and the motion data to obtain a geometric point cloud map of the target warehouse and the motion trajectory data of the robot. The recognition module is used to perform image recognition on the image data to determine the semantic information of the goods contained in the image; The fusion module is used to determine the target image data corresponding to the semantic information of each cargo, and to perform multimodal data fusion of the target image data, the geometric point cloud map and the motion trajectory data to generate the semantic point cloud map of the target warehouse. The generation module is used to perform feature analysis on the semantic point cloud map to generate an inventory map of the target warehouse; wherein the inventory map represents the inventory structure and inventory status of the target warehouse.

[0008] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the inventory map generation method described above are performed.

[0009] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the inventory map generation method described above.

[0010] This application provides a method, apparatus, electronic device, and storage medium for generating an inventory map. During the autonomous movement of a robot, multiple sensors acquire various data, which are then fused to create a precise geometric point cloud map of the warehouse. Semantic information about goods is extracted from the images. Through multimodal data fusion, the semantic information of the goods is associated with the geometric point cloud map to obtain a semantic point cloud map of the target warehouse. Finally, through feature analysis of the semantic point cloud map, an inventory map representing the inventory structure and status of the target warehouse is generated. This deep fusion of multi-sensor data solves the problem of insufficient information dimension from a single sensor, automatically constructing a high-precision inventory map within the warehouse that carries semantic information, forming a "digital twin" of the warehouse, and providing core data support for achieving refined and intelligent inventory management.

[0011] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a method for generating an inventory map according to an embodiment of this application is shown; Figure 2 This illustration shows a schematic diagram of the association between cargo semantic information and point cloud clusters provided in an embodiment of this application; Figure 3 This illustration shows a schematic diagram of a structured three-dimensional inventory map visualization provided in an embodiment of this application; Figure 4 This illustration shows a schematic diagram of the structure of an apparatus for generating an inventory map according to an embodiment of this application; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0015] First, the applicable application scenarios for this application will be introduced. This application can be applied to inventory management scenarios.

[0016] Research has found that with the rapid development of smart logistics and intelligent manufacturing, the digitalization of inventory management is the core of smart logistics and has become a key link in improving warehousing efficiency and reducing operating costs.

[0017] However, existing inventory information collection solutions for digital management mostly rely on single sensor modalities, leading to significant limitations in inventory management. For example, purely vision-based methods capture images of shelves with cameras, but their performance is heavily dependent on ambient lighting conditions, with recognition rates dropping significantly in dim, reflective, or low-texture scenes. Furthermore, this method struggles to obtain precise depth information and 3D dimensions of objects and exhibits poor robustness when faced with stacked or obstructed goods. Another example is LiDAR-based methods, which construct 3D point cloud models of the environment by emitting laser beams and measuring echo times. However, LiDAR cannot recognize semantic information; it only knows "there is something here," but not "what this thing is." Finally, RFID technology allows for contactless batch reading of tag information, but its spatial positioning accuracy is low, failing to provide information on item orientation, size, and specific placement, making it difficult to meet the needs of refined inventory management.

[0018] Furthermore, to overcome the insufficient information dimensionality of a single sensor, while multimodal sensor fusion technology exists, it is primarily applied to fields such as autonomous driving. Its algorithms and optimization objectives mainly focus on the detection and tracking of high-speed moving targets, such as real-time detection of pedestrians and vehicles. This is fundamentally different from the need for precise identification and measurement of densely packed, static goods in a warehouse environment. Therefore, current technologies still have gaps in areas specifically designed for inventory management scenarios, such as how to generate high-precision and information-rich inventory maps.

[0019] Based on this, embodiments of this application provide a method for generating an inventory map to automatically construct a high-precision inventory map within a warehouse that carries semantic information.

[0020] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for generating an inventory map according to an embodiment of this application. The method can be implemented using software and / or hardware; the inventory map generation system, implemented in hardware, can be mounted on a robot; the robot in this embodiment has autonomous mobility, and exemplarily, it can be a drone or an AGV (Automated Guided Vehicle). The robot includes a multimodal sensor suite, a data synchronization and calibration unit, and a processing unit; the multimodal sensor suite includes various sensors, such as at least one 3D LiDAR, at least one high-resolution camera, and at least one inertial measurement unit (IMU). The data synchronization and calibration unit performs strict time synchronization and spatial calibration on the data collected by each sensor in the multimodal sensor suite to ensure data consistency in a spatiotemporal reference. The processing unit internally runs a sensor fusion engine configured to perform multi-sensor data fusion to generate an inventory map of the target warehouse.

[0021] like Figure 1 As shown in the embodiments of this application, the generation method includes: S101. During the movement of the robot in the target warehouse, point cloud data, motion data of the robot, and image data of the target warehouse are acquired through various sensors carried by the robot.

[0022] In step S101, the robot autonomously navigates and moves within the target warehouse. During the movement, it acquires data along the way in real time through various sensors mounted on its body. Specifically, it can collect point cloud data of the surrounding area through the three-dimensional LiDAR on the robot; collect motion data of the robot through the inertial measurement unit (IMU) on the robot; and capture images of the shelves along the way through the camera on the robot to obtain image data of the target warehouse.

[0023] For example, an inertial measurement unit (IMU) is a core device that enables autonomous navigation by measuring the acceleration and angular velocity of a carrier. It consists of sensors such as gyroscopes and accelerometers. By calculating acceleration and angular velocity data in real time, it can continuously output information such as the three-dimensional attitude, displacement, and velocity of the carrier. In this embodiment, the IMU provides acceleration and angular velocity at a high frequency (e.g., 200Hz); the LiDAR provides 3D point clouds of the surrounding environment at a lower frequency (e.g., 10Hz); and the high-resolution camera captures images of the shelves along the route at a certain frequency (e.g., 30fps), or extracts images from a recorded video at a certain frequency. The image data includes the images and the acquisition time point corresponding to each frame of the image.

[0024] Accordingly, before acquiring data from the warehouse using the various sensors mounted on the robot, precise spatiotemporal calibration must be performed to ensure the consistency of data from various sensors on a spatiotemporal reference; therefore, the generation method also includes: The three-dimensional lidar, the inertial measurement unit, and the camera are synchronized in time, and the spatial transformation matrix between the radar coordinate system, the inertial coordinate system, and the camera coordinate system is calibrated.

[0025] For time synchronization, all sensors can achieve nanosecond-level time synchronization via hardware triggering or the Network Time Protocol (PTP). For spatial calibration (extrinsic parameter calibration), tools such as calibration boards can be used to accurately measure the relative rotation and translation relationships (i.e., the spatial transformation matrix) between the camera coordinate system, LiDAR coordinate system, and IMU coordinate system. This spatial transformation matrix serves as the mathematical basis for subsequent multimodal data fusion.

[0026] S102. Couple the point cloud data and the motion data to obtain the geometric point cloud map of the target warehouse and the motion trajectory data of the robot.

[0027] In this step, a tightly coupled method based on factor graph optimization (such as LIO-SAM) can be used to take the motion data of the IMU as the motion prior and use the point cloud data of LiDAR for registration and optimization, thereby generating a smooth, accurate and drift-free robot motion trajectory, and simultaneously constructing a global three-dimensional geometric point cloud map of the entire warehouse shelf.

[0028] S103. Perform image recognition on the image data to determine the semantic information of the goods contained in the image.

[0029] In this step, a pre-trained target detection model can be used to detect the image region where the cargo information identifier is located in each image of the image data; the decoding library is called to decode the image region where the cargo information identifier is located to obtain the cargo semantic information contained in the image.

[0030] For example, image data is analyzed in a visual pipeline manner, using a target detection model trained by deep learning (such as YOLO) to detect the image region containing the cargo information identifier in each image of the image data; for example, the cargo information identifier can be a barcode, QR code, or printed characters, etc. Then, a decoding library (such as ZBar) is called to decode the image region to obtain the cargo semantic information (such as cargo code: string "SN123456").

[0031] It should be noted that steps S102 and S103 can be executed simultaneously, or S102 can be executed first and then S103, or S103 can be executed first and then S102.

[0032] S104. For each cargo semantic information, determine the target image data corresponding to the cargo semantic information, and perform multimodal data fusion of the target image data, the geometric point cloud map and the motion trajectory data to generate the semantic point cloud map of the target warehouse.

[0033] In this step, by fusing the target image data corresponding to the semantic information of each item with the geometric point cloud map and motion trajectory data in a multimodal manner, the semantic information of the items can be spatially associated with the physical entities in the geometric point cloud map, generating a semantic point cloud map of the target warehouse containing the semantic information of the items. The physical entities can be a single box on a shelf or a stack of items; for example, the location of the semantic information of the items identified in the camera image can be back-projected into the geometric point cloud map constructed by LiDAR through pre-calibrated intrinsic and extrinsic parameters and combined with robot pose data to achieve spatial association.

[0034] In one possible implementation, step S104 may include: S1041. For each piece of cargo semantic information, determine the acquisition timestamp of the image corresponding to the cargo semantic information and the pixel coordinates of the image area where the cargo information identifier is located based on the image data, and obtain the target image data corresponding to the cargo semantic information.

[0035] As mentioned earlier, image data includes images and the acquisition time point corresponding to each frame. Therefore, for each cargo semantic information, the acquisition timestamp T of the corresponding image can be determined by searching the image data based on the image containing the cargo semantic information. In step S103, the target detection model will detect the image region where the cargo information identifier is located in each image of the image data, thereby determining the pixel coordinates. .

[0036] S1042. Based on the collection timestamp corresponding to the semantic information of the goods, extract the target point cloud data corresponding to the collection timestamp and extract the robot pose corresponding to the collection timestamp from the motion trajectory data.

[0037] Here, point cloud data and motion trajectory data are also organized according to a time point structure. Therefore, based on the acquisition timestamp, the corresponding target point cloud data can be extracted from the point cloud data, and the corresponding robot pose can be extracted from the motion trajectory data. ,in, Indicates the robot's position. This indicates the robot's posture.

[0038] S1043. Based on the pre-calibrated spatial transformation matrix and the robot pose corresponding to the acquisition timestamp, the pixel coordinates are back-projected onto the three-dimensional space of the geometric point cloud map to obtain the target ray.

[0039] In this step, the pixel coordinates can be first... A 3D ray is generated by backprojection based on the camera's intrinsic parameter K, represented as follows: Then, based on the robot's pose and spatial transformation matrix... The (external parameter matrix) transforms the 3D ray from the camera coordinate system to the world coordinate system of the 3D space of the geometric point cloud map, thus obtaining the target ray.

[0040] S1044. Determine the spatial intersection point between the target ray and the target point cloud data, and anchor the semantic information of the cargo to the point cloud cluster where the spatial intersection point is located in the target point cloud data to form the semantic point cloud map.

[0041] In this step, the spatial intersection points between the target ray and the target point cloud data are calculated, the point cloud clusters where the spatial intersection points are located are determined, and the semantic information of the cargo is anchored to these point cloud clusters. By anchoring the semantic information of each cargo to the corresponding point cloud clusters, the geometric point cloud map is transformed into a semantic point cloud map; wherein, the point cloud clusters where the spatial intersection points are located in the semantic point cloud map represent the three-dimensional point cloud representation of the cargo container corresponding to the semantic information of the cargo.

[0042] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the association between cargo semantic information and point cloud clusters, provided as an embodiment of this application. For example... Figure 2 As shown, the semantic information of the cargo (the string "SN123456") from the camera image perspective is anchored to the red 3D point cloud cluster from the LiDAR 3D point cloud perspective through data fusion.

[0043] S105. Perform feature analysis on the semantic point cloud map to generate an inventory map of the target warehouse.

[0044] The inventory map represents the inventory structure and inventory status of the target warehouse.

[0045] In one possible implementation, step S105 may include: S1051. Perform planar segmentation and point cloud clustering on the semantic point cloud map to identify the inventory structure of the target warehouse.

[0046] The inventory structure includes the shelving structure and the boundaries of each bin on each shelf. Using planar segmentation and clustering algorithms, the shelving structure (such as uprights and beams) and the bin boundaries are automatically identified from the map.

[0047] S1052. For each identified warehouse location, determine whether there is a point cloud cluster inside the warehouse location.

[0048] S1053. If it exists, determine that the status of the warehouse is occupied, record the semantic information of the goods anchored on the point cloud cluster, and determine the spatial scale of the point cloud cluster inside the warehouse.

[0049] S1054. If it exists, then the position is determined to be empty.

[0050] S1055. Determine the inventory status of the target warehouse based on the status of each warehouse location, the recorded semantic information of goods in the internal point cloud cluster, and the spatial scale of the internal point cloud cluster.

[0051] For steps S1052 to S1055, for each storage location, analyze whether there is a point cloud cluster inside it; if there is no point cloud inside, the status of the storage location is determined to be empty, and the data is represented as [Status: Empty]; if there is a point cloud inside, the status of the storage location is determined to be occupied, and the data is represented as [Status: Occupied]; and further calculate the spatial scale of the point cloud cluster inside the storage location. For example, the bounding box algorithm can be used to calculate the minimum bounding box of the point cloud cluster, and the data is represented as [Dimensions: Length, Width, Height]; if there is cargo semantic information anchored on the point cloud cluster, it is recorded together, and the data is represented as [ID: SN123456].

[0052] In this way, the status of each storage location on each shelf in the warehouse, the semantic information of the goods in the internal point cloud cluster, and the spatial scale of the internal point cloud cluster can be determined, thus obtaining the inventory status of the target warehouse.

[0053] S1056. Based on the inventory structure and inventory status of the target warehouse, a structured inventory map is formed.

[0054] Ultimately, the inventory structure and status of the target warehouse are organized into a structured inventory map, such as a structured database or JSON file, which can be directly accessed by upper-level systems such as WMS (Warehouse Management System).

[0055] Please see Figure 3 , Figure 3This is a schematic diagram illustrating a structured three-dimensional inventory map visualization method provided in an embodiment of this application. Figure 3 As shown, the inventory structure of this shelf is represented as 3. The shelving system has a total of 9 storage locations, 6 of which are occupied and 3 are vacant. The dimensions of the goods in each occupied location (spatial scale of point cloud clusters) are also shown. Combined with the structured data (JSON) example of storage location A-1-1, the 3D inventory map also includes the semantic information of the goods in that storage location, such as SN123456, as well as location information.

[0056] Furthermore, the generation method also includes: Based on the inventory map, an inventory quality analysis is performed on the target warehouse; wherein the analysis items include at least one of the following: warehouse occupancy rate, space utilization rate, whether the goods exceed the warehouse boundary, whether the goods packaging is abnormal, and whether the goods are stacked tilted.

[0057] In this step, based on the generated structured inventory map, further inventory quality analysis can be performed. For example, the warehouse occupancy rate can be calculated based on the ratio of occupied to vacant warehouse spaces; the space utilization rate can be calculated based on the size of the goods in the warehouse; whether the goods exceed the warehouse boundaries can be determined based on the size of the goods in the warehouse and the size of the warehouse; and whether the goods packaging is abnormal (damaged) and whether the goods are stacked at an angle can be determined based on the point cloud data of the goods in the inventory map. After any abnormality is detected in any analysis item, relevant personnel can be notified to handle it in a timely manner and / or the subsequent inventory scheduling strategy can be automatically optimized.

[0058] This application provides a method for generating an inventory map. During the autonomous movement of a robot, multiple sensors acquire various data, which are then fused to create a precise geometric point cloud map of the warehouse. Semantic information about goods is extracted from the images. Through multimodal data fusion, the semantic information of the goods is associated with the geometric point cloud map to obtain a semantic point cloud map of the target warehouse. Finally, through feature analysis of the semantic point cloud map, an inventory map representing the inventory structure and status of the target warehouse is generated. This deep fusion of multi-sensor data solves the problem of insufficient information dimension from a single sensor, automatically constructing a high-precision inventory map within the warehouse that carries semantic information, forming a "digital twin" of the warehouse, and providing core data support for achieving refined and intelligent inventory management.

[0059] Please see Figure 4 , Figure 4 This is a schematic diagram of a device for generating an inventory map, provided in an embodiment of this application. Figure 4 As shown, the generating apparatus 400 includes: The acquisition module 410 is used to acquire point cloud data, robot motion data, and image data of the target warehouse through multiple sensors mounted on the robot during the robot's movement in the target warehouse. The coupling module 420 is used to couple the point cloud data and the motion data to obtain a geometric point cloud map of the target warehouse and the motion trajectory data of the robot. The recognition module 430 is used to perform image recognition on the image data to determine the semantic information of the goods contained in the image; The fusion module 440 is used to determine the target image data corresponding to the semantic information of each cargo, and to perform multimodal data fusion of the target image data, the geometric point cloud map and the motion trajectory data to generate the semantic point cloud map of the target warehouse. The generation module 450 is used to perform feature analysis on the semantic point cloud map to generate an inventory map of the target warehouse; wherein the inventory map represents the inventory structure and inventory status of the target warehouse.

[0060] Furthermore, when the recognition module 430 performs image recognition on the image data to determine the semantic information of the goods contained in the image, the recognition module 430 is used to: The image region containing the cargo information identifier in each image of the image data is detected using a pre-trained object detection model. The decoding library is invoked to decode the image region where the cargo information identifier is located, thereby obtaining the cargo semantic information contained in the image.

[0061] Furthermore, when the fusion module 440 determines the target image data corresponding to each cargo semantic information and performs multimodal data fusion of the target image data, the geometric point cloud map, and the motion trajectory data to generate the semantic point cloud map of the target warehouse, the fusion module 440 is used to: For each piece of cargo semantic information, the acquisition timestamp of the image corresponding to the cargo semantic information and the pixel coordinates of the image area where the cargo information identifier is located are determined based on the image data to obtain the target image data corresponding to the cargo semantic information. Based on the collection timestamp corresponding to the semantic information of the goods, extract the target point cloud data corresponding to the collection timestamp and extract the robot pose corresponding to the collection timestamp from the motion trajectory data; Based on the pre-calibrated spatial transformation matrix and the robot pose corresponding to the acquisition timestamp, the pixel coordinates are back-projected onto the three-dimensional space of the geometric point cloud map to obtain the target ray; The spatial intersection point between the target ray and the target point cloud data is determined, and the semantic information of the cargo is anchored to the point cloud cluster in the target point cloud data where the spatial intersection point is located, thus forming the semantic point cloud map; wherein, the point cloud cluster in the semantic point cloud map where the spatial intersection point is located represents the three-dimensional point cloud representation of the cargo box corresponding to the semantic information of the cargo.

[0062] Furthermore, when the generation module 450 performs feature analysis on the semantic point cloud map to generate the inventory map of the target warehouse, the generation module 450 is used to: The semantic point cloud map is subjected to planar segmentation and point cloud clustering to identify the inventory structure of the target warehouse; wherein, the inventory structure includes the shelf structure and the boundary of each storage location on each shelf; For each identified warehouse location, determine whether a point cloud cluster exists within that warehouse location; If it exists, the status of the warehouse is determined to be occupied, the semantic information of the goods anchored on the point cloud cluster is recorded, and the spatial scale of the point cloud cluster inside the warehouse is determined. If it exists, then the position is determined to be empty; The inventory status of the target warehouse is determined based on the status of each warehouse location, the recorded semantic information of goods in the internal point cloud cluster, and the spatial scale of the internal point cloud cluster. Based on the inventory structure and inventory status of the target warehouse, a structured inventory map is formed.

[0063] Furthermore, the generating apparatus 400 further includes: an analysis module; the analysis module is used for: Based on the inventory map, an inventory quality analysis is performed on the target warehouse; wherein the analysis items include at least one of the following: warehouse occupancy rate, space utilization rate, whether the goods exceed the warehouse boundary, whether the goods packaging is abnormal, and whether the goods are stacked tilted.

[0064] Furthermore, when the acquisition module 410 acquires point cloud data, robot motion data, and image data within the target warehouse through various sensors mounted on the robot during the robot's movement in the target warehouse, the acquisition module 410 is used for: The robot collects point cloud data of the surrounding area using a 3D LiDAR. The robot's motion data is collected by the inertial measurement unit mounted on the robot. The robot uses a camera to capture images of the shelves along the route, thus obtaining image data of the target warehouse.

[0065] Furthermore, the generating apparatus 400 further includes: a synchronization module; the synchronization module is used for: The three-dimensional lidar, the inertial measurement unit, and the camera are synchronized in time, and the spatial transformation matrix between the radar coordinate system, the inertial coordinate system, and the camera coordinate system is calibrated.

[0066] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 includes a processor 510, a memory 520, and a bus 530.

[0067] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 and the memory 520 communicate via the bus 530. When the machine-readable instructions are executed by the processor 510, the steps of the inventory map generation method in the above method embodiment can be performed. For specific implementation details, please refer to the method embodiment, which will not be repeated here.

[0068] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it can execute the steps of the inventory map generation method as described in the above method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0069] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0071] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0072] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0073] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of generating an inventory map, characterized by, The generating method comprises: In the movement of the robot in the target warehouse, acquiring point cloud data in the target warehouse, motion data of the robot and image data in the target warehouse by various sensors carried by the robot; Coupling the point cloud data and the motion data to obtain a geometric point cloud map in the target warehouse and motion trajectory data of the robot; Performing image recognition on the image data to determine the semantic information of the goods contained in the image; For each semantic information of the goods, determining target image data corresponding to the semantic information of the goods, and performing multi-modal data fusion on the target image data, the geometric point cloud map and the motion trajectory data to generate a semantic point cloud map of the target warehouse; Performing feature analysis on the semantic point cloud map to generate an inventory map of the target warehouse; wherein the inventory map represents the inventory structure and the inventory state of the target warehouse.

2. The generation method of claim 1, wherein, The image recognition on the image data to determine the semantic information of the goods contained in the image comprises: Using a pre-trained target detection model to detect the image area where the goods information identifier is located in each image in the image data; Calling a decoding library to decode the image area where the goods information identifier is located to obtain the semantic information of the goods contained in the image.

3. The generation method of claim 2, wherein, For each semantic information of the goods, determining target image data corresponding to the semantic information of the goods, and performing multi-modal data fusion on the target image data, the geometric point cloud map and the motion trajectory data to generate a semantic point cloud map of the target warehouse, comprises: For each semantic information of the goods, determining the acquisition time stamp of the image corresponding to the semantic information of the goods and the pixel coordinates of the image area where the goods information identifier is located according to the image data to obtain the target image data corresponding to the semantic information of the goods; According to the acquisition time stamp corresponding to the semantic information of the goods, extracting target point cloud data corresponding to the acquisition time stamp and extracting the robot pose corresponding to the acquisition time stamp from the motion trajectory data; Based on the pre-calibrated spatial transformation matrix and the robot pose corresponding to the acquisition time stamp, the pixel coordinates are back-projected to the three-dimensional space of the geometric point cloud map to obtain a target ray; Determining the spatial intersection between the target ray and the target point cloud data, anchoring the semantic information of the goods to the point cloud cluster where the spatial intersection is located in the target point cloud data to form the semantic point cloud map; wherein the point cloud cluster where the spatial intersection is located in the semantic point cloud map represents the three-dimensional point cloud representation of the goods box corresponding to the semantic information of the goods.

4. The generation method of claim 3, wherein, The feature analysis on the semantic point cloud map to generate an inventory map of the target warehouse comprises: Performing plane segmentation and point cloud clustering on the semantic point cloud map to identify the inventory structure of the target warehouse; wherein the inventory structure comprises the shelf structure and the boundary of each storage position on each shelf; For each identified storage position, determining whether there is a point cloud cluster inside the storage position; If there is, determining that the state of the storage position is occupied, recording the semantic information of the goods anchored on the point cloud cluster, and determining the spatial scale of the point cloud cluster inside the storage position; If exists, determine the state of the position as empty; According to the state of each position, the recorded internal point cloud cluster cargo semantic information and the spatial scale of the internal point cloud cluster, the inventory state of the target warehouse is determined; According to the inventory structure and inventory state of the target warehouse, a structured inventory map is formed.

5. The generation method of claim 4, wherein, The generation method further comprises: Based on the inventory map, the target warehouse is analyzed for inventory quality; wherein the analysis items include at least one of the following: position occupancy rate, space utilization rate, whether the goods exceed the position boundary, whether the goods packaging is abnormal, and goods stacking inclination.

6. The generation method of claim 1, wherein, Through the multiple sensors carried by the robot, point cloud data, motion data of the robot and image data in the target warehouse are obtained, including: Through the three-dimensional laser radar carried by the robot, the surrounding point cloud data is collected; Through the inertial measurement unit carried by the robot, the motion data of the robot is collected; Through the camera carried by the robot, the image of the passing shelf is taken to obtain the image data in the target warehouse.

7. The generation method of claim 6, wherein, Before the point cloud data, motion data of the robot and image data in the target warehouse are obtained through the multiple sensors carried by the robot, the generation method further comprises: The three-dimensional laser radar, the inertial measurement unit and the camera are time-synchronized, and the spatial transformation matrix between the radar coordinate system, the inertial coordinate system and the camera coordinate system is calibrated.

8. An apparatus for generating an inventory map, characterized by comprising: The generation device comprises: An acquisition module is configured to, during movement of a robot in a target warehouse, acquire, through multiple sensors carried by the robot, point cloud data in the target warehouse, motion data of the robot and image data in the target warehouse; A coupling module is configured to couple the point cloud data and the motion data to obtain a geometric point cloud map in the target warehouse and motion trajectory data of the robot; An identification module is configured to perform image recognition on the image data to determine cargo semantic information contained in the image; A fusion module is configured to, for each cargo semantic information, determine target image data corresponding to the cargo semantic information, and perform multi-modal data fusion on the target image data, the geometric point cloud map and the motion trajectory data to generate a semantic point cloud map of the target warehouse; A generation module is configured to perform feature analysis on the semantic point cloud map to generate an inventory map of the target warehouse; wherein the inventory map represents the inventory structure and inventory state of the target warehouse.

9. An electronic device, comprising: It comprises: A processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, the machine readable instructions are executed by the processor to perform the steps of the inventory map generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, which is executed by the processor to perform the steps of the inventory map generation method according to any one of claims 1 to 7.

Images