Goods inventory-taking method and device

Abstract

To provide a method and device for realizing automatic inventory of cargoes by programmed automatic-control, thereby improving the efficiency of cargo inventory.SOLUTION: A method includes: obtaining a location image to be obtained by image collection for at least one location for storing cargoes; obtaining a detection result by performing inference detection on the location image; and obtaining a result of cargo inventory on the location by emphasizing cargoes in the location image based on the detection result. The detection result can be produced automatically by performing inference detection on the location image. Based on the detection result produced automatically, the cargoes in the location image are automatically emphasized.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] This application relates to the technical field of logistics applications, and particularly to a method and apparatus for inventorying goods.

Background Art

[0002] With the development of the logistics industry, the demand for goods is increasing. In many scenarios of goods handling, since it involves the inventory of goods, it is very important to improve the efficiency of goods inventory.

[0003] Currently, the inventory of goods is mainly carried out manually, but the efficiency of such inventory is low.

Summary of the Invention

[0004] In a first aspect, this application provides a method for inventorying goods. The method includes the following. Obtain a location image obtained by collecting images for at least one location for storing goods. Obtain a detection result by performing estimated detection on the location image. Obtain the inventory result of the goods at the location by highlighting the goods in the location image based on the detection result.

[0005] In a second aspect, this application further provides an apparatus for inventorying goods. The apparatus includes a processor and a memory. A computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the following. Obtain a location image obtained by collecting images for at least one location for storing goods. Obtain a detection result by performing estimated detection on the location image. Obtain the inventory result of the goods at the location by highlighting the goods in the location image based on the detection result.

[0006] According to the above-described method and apparatus for inventorying goods, a location image obtained by collecting an image for at least one location for storing goods is acquired. A detection result is obtained by presumptively detecting the location image. An inventory result of the goods at the location is obtained by highlighting the goods in the location image based on the detection result. In the present application, by presumptively detecting the location image, the detection result can be automatically generated. Based on the automatically generated detection result, the goods in the location image are automatically highlighted, and automatic inventory of the goods is realized by program automatic control, and the inventory efficiency of the goods can be improved.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0008] To make the objectives, technical solutions, and advantages of the present application clearer, the present application will be described in more detail below with reference to the drawings and embodiments. The specific embodiments described in this specification are only used to explain the present application and are not used to limit the present application.

[0009] The method for inventorying goods according to the embodiments of the present application can be applied to the application environment shown in FIG. 1. As shown in FIG. 1, it is necessary that the server 104 is connected to the same network as the image collection device 102 on the side opposite to the image collection device 102 so as to ensure that the server 104 can access all the image collection devices 102 for image collection. The server 104 communicates with the image collection device 102 via the network. The image collection device 102 acquires a location image (also referred to as a storage area map) by performing image collection on the location for storing goods. The image collection device 102 can transmit the collected location image to the server 104, whereby the server 104 performs inventorying of goods. Specifically, after the server 104 acquires the location image, the server 104 acquires a detection result by performing estimation detection on the location image with an estimation detection engine integrated and deployed on the server 104. The estimation detection engine is obtained by serializing the model parameters of a goods detection model trained by deep learning, and the estimation detection speed of the estimation detection engine is faster than the estimation detection speed of the goods detection model. Finally, based on the detection result, the server 104 highlights the goods in the location image, thereby obtaining the inventory result of the goods in the location. The number of image collection devices 102 may be one or a plurality. The server 104 may be an independent server or a server cluster composed of a plurality of servers.

[0010] In some embodiments, a method for inventorying goods is provided. The method can be applied to a server and can also be realized through the interaction between the server and the image collection device. In the embodiments of the present application, it is not specifically limited thereto. As shown in FIG. 2, by way of example, the method is applied to the server in FIG. 1 and includes the following steps.

[0011] Step 202: Obtain a location image obtained by performing image collection on at least one location for storing goods.

[0012] The location image of this application is obtained by the image collection device performing image collection. The image collection device refers to a device equipped with a photographing function, which may be various cameras, or a mobile device incorporating a photographing device such as a camera, and is not limited thereto.

[0013] A location refers to an area within a warehouse for storing goods, and the shape of the location is generally rectangular. A location image is an image obtained by the image collection device performing image collection on a specific location or multiple locations. One or more locations may be displayed in the location image, and the status of the goods at each location may also be displayed. Note that the location image can further include the image content of other areas such as a security area in addition to the image content of the location. If there is the image content of the location in the location image, the requirements of the technical solution of the present invention can be satisfied.

[0014] The security area refers to the area around the location and is used for warning prompts. For example, when it is identified that a person or a forklift has entered the security area range, a warning prompt is generated. Note that the shape of the security area may be polygonal, and the number of vertices, area, and position of the security area can all be set according to actual needs.

[0015] Specifically, the server acquires the location image of the location for storing goods collected by the image collection device. The location image is used for subsequent estimation detection.

[0016] Step 204: Obtain a detection result by performing estimation detection on the location image.

[0017] The detection result is obtained by estimating and detecting the location image with an estimation detection engine. The estimation detection engine may be integrated and deployed on a server. The estimation detection engine is obtained by serializing the model parameters of a cargo detection model trained by deep learning. The estimation detection speed of the estimation detection engine is faster than that of the cargo detection model. The cargo detection model established by the deep learning algorithm can automatically find the most accurate features, thereby increasing the accuracy rate of cargo detection. By converting the cargo detection model trained by deep learning into an estimation detection engine, the estimation can be further optimized and the accuracy rate of cargo detection can be further increased.

[0018] Deep learning refers to learning the internal rules and expression levels of sample data, and during such learning, data such as characters, images, and voices can be identified.

[0019] The cargo detection model refers to a target detection algorithm model trained by a deep learning algorithm and is used for the detection and identification of cargo. Note that the cargo detection model may be a multi-classification model, that is, the cargo detection model may not only be used for the detection and identification of cargo, but may also recognize objects such as humans and forklifts.

[0020] The estimation detection engine refers to an estimation optimization tool that optimizes a trained cargo detection model. The estimation detection engine can optimize the estimation, thereby making the identification of various cargos more accurate and realizing the acceleration of deployment.

[0021] The process of converting a cargo detection model into an estimation detection engine is as follows. Convert the format of the model file corresponding to the trained cargo detection model into the format of the file corresponding to the estimation detection engine to generate an engine file. Specifically, by introducing the cargo detection model into the estimation detection engine, an engine file is generated, and the engine file is serialized and saved, whereby the saved engine file can be easily and quickly called to accelerate the estimation of the cargo detection model.

[0022] In actual applications, tensorrt can be selected as the estimation detection engine of this application, and the yolov5 model can be selected as the cargo detection model of this application. Tensorrt can be an estimation framework in C++ that can operate on various hardware platforms with a Graphics Processing Unit (GPU). The yolov5 model is a neural network model constructed using a single-stage target detection algorithm.

[0023] Based on the above, the process of converting a cargo detection model into an estimation detection engine can further include the following. Convert the model file generated by the trained cargo detection model into a TensorRT engine file, that is, convert the model file with the.pt format into a TensorRT engine file with the.engine format. The reason for converting as above is that the software system of this application utilizes the deployment scheme of TensorRT, which is faster than the conventional scheme and relatively less resource-consuming. The principle of TensorRT is to obtain TensorRT by serializing and converting the model parameters of the cargo detection model, and then directly input the subsequent input data into the serialized TensorRT for estimation. Therefore, first, use the TensorRT developer kit to define the YOLOv5 model. Specifically, the YOLOv5 model can correspond to the network structure of version v5.0. Next, for subsequent use of the software, convert the model file with the.pt format into a TensorRT engine file with the.engine format.

[0024] Specifically, in the estimation detection engine integrated and deployed on the server, estimate and detect the cargo in the location image collected by the image collection device to obtain the detection result. The detection result includes the identified location and the status of the cargo at that location (such as the type of cargo and the number of cargo, etc.).

[0025] Note that the estimation detection engine can not only identify and detect locations and goods, but also further identify security areas, humans, and forklifts, and obtain corresponding detection results. Specifically, the detection results include the identified location and the status of the goods at that location (such as the type of goods and the number of goods). The detection results further include whether there are humans or forklifts in non-location areas (such as security areas). When it is detected that there are humans or forklifts in the security area, a warning is generated, for example, a warning prompt is generated.

[0026] Step 206: Obtain the inventory result of the goods at the location by highlighting the goods in the location image based on the detection results.

[0027] Highlighting the goods means highlighting the area position of the goods in the location image. Through highlighting, the user can more clearly recognize the specific position of the goods in the location image. Highlighting includes, but is not limited to, highlighting the goods, filling in colors and text annotations, and drawing area detection blocks corresponding to the goods based on the recognized goods.

[0028] Specifically, based on the detection results, by highlighting the goods in the location image, the inventory result of the goods at the location is obtained, that is, the type of goods and the number of goods at each location corresponding to the location image are obtained. Note that the server can display the location image with the goods highlighted on the display device.

[0029] In the above-described method for inventorying goods, a location image is acquired, a detection result is obtained by estimating and detecting the location image, and the inventory result of the goods at the location is obtained by highlighting the goods in the location image based on the detection result. In the present application, the detection result can be automatically generated by estimating and detecting the location image. Based on the automatically generated detection result, the goods in the location image are automatically highlighted, and automatic inventorying of the goods is realized by program automatic control, and the inventory efficiency of the goods can be improved.

[0030] In some embodiments, before step 204, the method further includes the step of training a goods detection model. Specifically, this step includes the following. A training image set is formed by capturing a process image of the goods in transit at the location by an image collection device corresponding to the location. A goods detection model is obtained by training an original detection model based on the training image set.

[0031] The image collection device needs to be installed at an appropriate position. For example, by installing the image collection device on both opposite sides of the location, it is ensured that the goods at the location are completely within the shooting field of view of the image collection device. In addition, when stacking of goods is required, it is necessary to ensure that all stacked goods are within the shooting field of view of the image collection device. Also, the original detection model is an initial neural network model to be trained.

[0032] Specifically, to ensure that the server can access all image collection devices, the installed image collection devices and the corresponding servers are connected to the same network. By turning on the image collection function of the image collection devices by the server, a training image set for training is obtained by capturing the entire process of cargo transportation (i.e., process images) at locations within the visual field range of each image collection device. Next, an annotated image set is obtained by annotating each process image in the training image set. The types of annotations include at least one of humans, forklifts, and goods. By inputting the annotated annotated image set into the original detection model for training, optimal model parameters are obtained. By updating the original detection model based on the optimal model parameters, a trained cargo detection model is obtained. As the original detection model, a yolov5 model can be selected. The cargo detection model obtained by training based on the yolov5 model has a high estimation speed, a small occupied space, and higher accuracy.

[0033] In some embodiments, after turning on the image collection function of the image collection device, the server can further collect images by calling the image capture function in the Software Development Kit (SDK) developed depending on the image collection device. The SDK is generally an aggregate of development tools when some software engineers build application software for specific software packages, software frameworks, hardware platforms, operating systems, etc.

[0034] In some embodiments, step 202 specifically includes, but is not limited to, the following. Obtain the location images collected by the image collection device and store the location images in the image buffer.

[0035] The image buffer is used to cache the images captured by the image collection device corresponding to the thread.

[0036] Specifically, the image data collected by the image collection device can be input into the server via a network cable. The server turns on one thread for each image collection device, calls the image capture function in the thread, extracts the location image in the video streaming collected by the image collection device, decodes the location image, and stores it in the image buffer.

[0037] Note that the decoded image is stored in the vector container in the memory as a Mat object in OpenCV (open source computer vision library), and is automatically released after the processing is completed. OpenCV is a cross-platform software library for computer vision and machine learning. The Mat object is a memory object in OpenCV for storing image information, and can be understood as a pixel matrix containing all intensity values. The vector container is a sequence container encapsulating a dynamically sized array.

[0038] In some embodiments, step 204 specifically includes, but is not limited to, the following. Wake up the cargo estimation thread, obtain the location image from the image buffer in the cargo estimation thread, and call the estimation detection engine to perform estimation detection on the location image, thereby obtaining the detection result.

[0039] The cargo estimation thread is used to estimate the location image in the image buffer.

[0040] Specifically, if there is an image in the image buffer, the cargo estimation thread is woken up. Then, the cargo estimation thread acquires a location image from the image buffer and calls the estimation detection engine to perform estimation detection on the location image, thereby obtaining a detection result. The detection result includes the identified location and the status of the cargo at that location (e.g., the type of cargo and the quantity of cargo, etc.).

[0041] In some embodiments, the location image is acquired by an image collection thread that is turned on towards the image collection device. In this case, the method for taking inventory of the cargo specifically includes, but is not limited to, the following. When the number of images stored in the image buffer reaches a preset threshold, the image collection device is controlled via the image collection thread to stop image collection within a preset time range. For example, a notice to temporarily stop the collection can be sent to the image collection thread.

[0042] Specifically, the number of images stored in the image buffer is obtained and used as the number of stored images. When the number of stored images reaches a preset threshold, that is, when the number of stored images reaches the upper limit of the buffer capacity and no additional images can be stored in the image buffer, a notice to temporarily stop the collection is sent to the image collection thread, and the image collection device is notified via the image collection thread to stop image collection within a preset time range (e.g., within 10 milliseconds). When the number of stored images has not reached the preset threshold, the image collection device can perform image collection normally.

[0043] Note that the image buffer is arranged because in the actual application, the speed at which the image collection device captures images does not match the speed at which the cargo estimation thread performs estimations. For easier understanding, an example will be given for explanation. Assume that the time for the image collection device to capture one image is approximately 50 milliseconds (the specific capture time varies depending on the model number of the image collection device), and the time for the cargo estimation thread to estimate one image is approximately 10 milliseconds (the specific estimation speed varies depending on the model number of the graphics card). Assume that 10 image collection devices are operating simultaneously, and since the times for the image collection devices to capture images are synchronized, 10 images will be captured within 50 milliseconds. However, since the cargo estimation thread can only process 5 images within 50 milliseconds, unprocessed images will remain. If the image collection device continues to capture images, a large number of images will accumulate. Considering the above situation, in the present application, an image buffer is arranged, and the images captured by the image collection device are stored in the image buffer by the image collection thread. If there are images in the image buffer, the image buffer continuously notifies the cargo estimation thread to perform estimation detection. When there is no capacity in the image buffer, the image buffer notifies the image collection thread to stop capturing images. Thereby, the speed at which the image collection device captures images is made to match the speed at which the cargo estimation thread performs estimations.

[0044] In some embodiments, the cargo estimation thread can call the estimation detection engine to estimate and detect the location image, and after obtaining the detection result, save the detection result in the estimation buffer. The estimation buffer is used to cache the cargo estimation thread and the cargo display thread. If there are detection results and corresponding location images in the estimation buffer, the cargo display thread is woken up. After the cargo display thread captures the detection result and the corresponding location image (i.e., the original drawing, the drawing before estimation detection) from the estimation buffer, the detection result is drawn and displayed on the original drawing. Specifically, the number of cargos at the location can be displayed.

[0045] Note that the image collection thread, the cargo estimation thread, and the cargo display thread in the embodiments of the present application are each independent threads, and in the embodiments of the present application, one thread is arranged for each image collection device. Thereby, it is ensured that images of each image collection device can be acquired simultaneously.

[0046] In some embodiments, step 204 includes, but is not limited to, the following. By detecting at least one location corresponding to the location image, a target location is acquired. By detecting the cargo at the target location with an estimation detection engine, a detection result is acquired.

[0047] The target location is a location target for detecting the cargo stored therein, identified from the location image. By detecting the cargo at the target location with an estimation detection engine, a detection result is acquired. The detection result includes the identified location and the state of the cargo at the location (for example, the position of the cargo, the type of the cargo, the number of the cargo, etc.).

[0048] In some embodiments, the step of "acquiring a target location by detecting at least one location corresponding to the location image" includes, but is not limited to, the following. Determining an image collection device corresponding to the location image (which may be referred to as a target image collection device here). Acquiring a pre-configuration file and acquiring location information corresponding to the target image collection device from the pre-configuration file. By positioning at least one location corresponding to the location image based on the location information, a target location is acquired.

[0049] The target image collection device refers to the image collection device that captured the location image. The location information is acquired by calibrating a sample location image, and the sample location image is acquired by the target image collection device performing image collection on at least one location in advance.

[0050] Specifically, calibration is performed on the acquired image. First, the acquired image is opened with a drawing tool, and the length and width of the location are calculated based on the pixel values (X, Y) at the four corners of the location, thereby performing a location plan. The location configuration profile is opened, the pixel values, length, and width of the planned location are input into the configuration profile, the visualization program is opened, the state of the location (for example, whether the goods exceed the boundary of the location) is observed, and it is ensured that the goods state of each location is updated in real time.

[0051] The location information includes, but is not limited to, the spatial size of the location (for example, the length and width of the location), the location number, and the specific position of the location in the location image. The specific position of the location in the location image refers to the pixel coordinates of each vertex of the location in the pixel coordinate system.

[0052] Note that one reference image for calibrating the location is selected from the set of reference images captured by each image collection device. The selection criterion for the reference image is to select an image in which the location completely captured within the field of view of the image collection device and the features are complete and clear. When the reference image is used to calibrate the security area for alarm, an image that simultaneously includes the complete location and the security area within the field of view of the image collection device and the features are complete and clear must be selected. After an appropriate reference image is selected, the calibration method is to use a drawing tool or a calibration tool to draw the vertices of each location and the vertices of the security area on the reference image, and write the pixel coordinates of each drawn vertex into a pre-configuration file. The shape of the location is generally rectangular, and the number of corresponding vertices is four. The shape of the security area may be polygonal, and the specific number of vertices can be adjusted according to actual needs. For the format of the pre-configuration file, reference can be made to the format of the configuration file configured in the inventory software, for example, the xml format can be cited.

[0053] Note that the pre-configuration file is used to set the specific position area of the location and the security area. The position data of both of the above are defined by the above steps. After the image collection device is arranged on site, since the position of the image collection device in the warehouse is fixed, the pre-configuration file is used to draw the location and the security area on the location image captured by the image collection device to represent the positions of the location and the security area in the location image.

[0054] Specifically, after determining the target image acquisition device corresponding to the location image, a pre-configuration file is obtained, and then the specific position of the location in the location image (i.e., the pixel coordinates of each vertex of the location recorded in the pre-configuration file) is obtained from the pre-configuration file. By calibrating the pixel coordinates of each vertex in the location information to the location image, the target location is obtained.

[0055] In some embodiments, the inventory result of the goods includes the number of goods at the location. Step 206 specifically includes, but is not limited to, the following. Based on the detection result, a first detection block for positioning the goods is drawn on the location image. A second detection block in the location image for positioning the target location is determined. Based on the degree of overlap between the first detection block and the second detection block, the number of goods at the target location is determined.

[0056] Specifically, based on the detection result, the specific position of each good at the location in the location image can be determined. Also, based on the specific position of each good in the location image, a first detection block for positioning the goods can be drawn. For example, based on the pixel coordinates of each vertex of each good, the first detection block can be drawn on the location image. Based on the pixel coordinates of each vertex corresponding to the target location, a second detection block for positioning the target location can be drawn on the location image. The first detection block can be represented by vertex coordinates, length, and width, and the first detection block can be a polygonal block, specifically, a rectangular block may be selected. The second detection block can also be represented by vertex coordinates, length, and width, and the second detection block can be a polygonal block, specifically, a rectangular block may be selected.

[0057] Specifically, the overlapping degree between the first detection block and the second detection block is calculated. When the overlapping degree is equal to or greater than a preset overlapping range, it is determined that the first detection block is located within the second detection block, and it is further determined that the cargo corresponding to the first detection block is located at the target location corresponding to the second detection block. When the overlapping degree is less than the preset overlapping range, it is determined that the first detection block is not located within the second detection block, and it is further determined that the cargo corresponding to the first detection block is not located at the target location corresponding to the second detection block. The number of cargos at the target location is determined. Specifically, the number of first detection blocks whose overlapping degree with the second detection block is equal to or greater than the preset overlapping range is determined.

[0058] For ease of understanding, an example is given for explanation. Assume that the preset overlapping range is 40%, the first detection block includes detection block A and detection block B, detection block A corresponds to cargo A, and detection block B corresponds to cargo B. When the overlapping range between detection block A and the second detection block is 50%, which is greater than the preset overlapping range, it is explained that detection block A is located within the second detection block, that is, cargo A is located at the target location. When the overlapping range between detection block B and the second detection block is 20%, which is less than the preset overlapping range, it is explained that detection block B is not located within the second detection block, that is, cargo B is not located at the target location. According to the above, it can be determined that only cargo A is at the target location and the number of cargos is 1.

[0059] In some embodiments, since the pixel coordinates of each vertex of the security area are calibrated in the pre-configuration file, the third detection block in the location image for positioning the security area can be similarly determined. A human object or a forklift object within the third detection block can be identified by the estimation detection engine.

[0060] In some embodiments, as shown in FIG. 3, before executing the software capable of inventorying goods, it is necessary to attach the image collection device at an appropriate height of the platform to ensure that all goods at the location are completely displayed within the shooting field of view of the image collection device. To ensure that the server can access all image collection devices, the image collection device and the corresponding server are connected to the same network. After attaching the image collection device, it is necessary to collect images. Specifically, it is necessary to capture the entire process of goods transportation within the field of view of each image collection device, and an image set is obtained by annotating the collected images. By putting the annotated image set into a neural network model for training, a model file with a.pt format is obtained, the.pt file is converted into an.engine file, and a converted estimation detection engine is obtained. Also, one image captured by each image collection device is selected and used to calibrate the location and security area to obtain a configuration file. Next, the arranged configuration file and the converted estimation detection engine are put into the software, that is, the arranged configuration file and the converted estimation detection engine are put into the directory of the folder where the software is located, and the software can be executed by clicking on the software icon.

[0061] Note that one location area contains multiple locations. Multiple image acquisition devices for image acquisition can be installed so that the goods at all locations in the location area can be detected. Each image acquisition device can simultaneously photograph one or more locations and the goods corresponding to the locations, and the actual number of image acquisition devices can be determined based on the number of locations at the site. By simultaneously photographing the locations in the location area with multiple image acquisition devices, multiple location images that can cover the entire location area can be obtained. By detecting the goods in the multiple location images with one industrial computer, more comprehensive detection results can be obtained, and the accuracy rate of inventory counting can be increased.

[0062] At the same site, after configuring the configuration file and model, it is not necessary to repeatedly execute the steps of file configuration and model training. At different sites, it is necessary to reconfigure the file and retrain the model.

[0063] In some embodiments, as shown in FIG. 4, when the above software is executed, the data collected by the image acquisition device is input into the server via the network cable. The server turns on one thread for each image acquisition device, turns on the image capture function, decodes the captured image, and stores it in the image buffer. When there is an image in the image buffer, the goods estimation thread (i.e., the tensorrt estimation thread) is woken up to perform estimation detection. Each image estimated and detected by the goods estimation thread is stored in the next intermediate container (i.e., the estimation buffer). When there is an image in the estimation buffer, the display thread is woken up, and the estimation result is drawn and displayed on the original drawing by the display thread.

[0064] Note that although each step in the flowchart according to each of the above embodiments is described sequentially according to the arrows, it should be understood that these steps are not necessarily executed sequentially according to the arrows. Unless otherwise specified in this specification, the execution of these steps is not strictly limited to that order, and these steps may be executed in other orders. Furthermore, at least a part of the steps in the flowchart according to each of the above embodiments may include a plurality of steps or stages, and these steps or stages are not necessarily executed and completed simultaneously, and may be executed at different timings. Also, the execution of these steps or stages is not necessarily continuous, and may be executed in order or alternately with at least a part of other steps or steps and stages in other steps.

[0065] Based on the same inventive concept, in an embodiment of the present application, a cargo inventory device for implementing the above-described cargo inventory method is further provided. Since the solution to the problems related to the device is the same as the solution described in the above cargo inventory method, for the specific limitations of one or more embodiments of the following cargo inventory device, reference can be made to the limitations on the above cargo inventory method, and details will not be described here.

[0066] In one embodiment, as shown in FIG. 5, a cargo inventory device is provided. The cargo inventory device includes an image acquisition module 502, an estimation detection module 504, and a cargo inventory module 506.

[0067] The image acquisition module 502 is configured to acquire a location image. The location image is acquired by collecting an image of the location for storing the cargo. The location image of the present application is acquired by an image collection device performing image collection. The image collection device refers to a device having a photographing function, which may be various cameras, or a mobile device having a photographing device such as a camera built therein, and is not limited thereto.

[0068] As one embodiment, the image acquisition module 502 may be connected to the image collection device by wire or wirelessly, may be integrated into the image collection device, or may be integrally formed with the image collection device. In the present invention, it is not limited thereto.

[0069] The estimation detection module 504 is configured to obtain a detection result by estimating and detecting a location image.

[0070] The inventory module 506 is configured to obtain an inventory result of the goods at the location by highlighting the goods in the location image based on the detection result.

[0071] In the present application, according to the above-described goods inventory device, a location image obtained by performing image collection on at least one location for storing goods is acquired. A detection result is obtained by estimating and detecting the location image. An inventory result of the goods at the location is obtained by highlighting the goods in the location image based on the detection result. By estimating and detecting the location image, the detection result can be automatically generated. Based on the automatically generated detection result, the goods in the location image can also be automatically highlighted. In the present application, automatic inventory of goods is realized by program automatic control, and the inventory efficiency of goods can be improved.

[0072] In some embodiments, the detection result is obtained by estimating and detecting a location image with an estimation detection engine. The estimation detection engine is obtained by serializing the model parameters of a goods detection model trained by deep learning. The estimation detection speed of the estimation detection engine is faster than the estimation detection speed of the goods detection model.

[0073] In some embodiments, the image acquisition module 502 is further configured to acquire the location image collected by the image collection device and store the location image in the image buffer. The estimation detection module 504 is further configured to wake up the cargo estimation thread to acquire the location image from the image buffer and call the estimation detection engine to perform estimation detection on the location image, thereby obtaining a detection result.

[0074] In some embodiments, the cargo inventory device further includes an image buffer module. When the number of images stored in the image buffer reaches a preset threshold, the image buffer module controls, via the image collection thread, at least one image collection device to stop image collection within a preset time range.

[0075] In some embodiments, the estimation detection module 504 includes a location detection unit and a cargo detection unit. The location detection unit is configured to obtain a target location by detecting at least one location corresponding to the location image (abbreviated as location detection). The cargo detection unit is configured to obtain a detection result by detecting the cargo at the target location with the estimation detection engine.

[0076] In some embodiments, the estimation detection module 504 further includes a security area identification unit. The security area identification unit is configured to identify the object and / or the number of objects within the security area. For example, it may identify the humans and the number of humans within the security area, or it may identify the forklifts and the number of forklifts within the security area. When the security area identification unit identifies that a human or a forklift has entered the security area, it generates a warning and serves as a warning prompt.

[0077] In some embodiments, the location detection unit is further configured to obtain a target location by determining an image acquisition device corresponding to a location image, obtaining a pre-configuration file, obtaining location information corresponding to the image acquisition device from the pre-configuration file, and positioning at least one location corresponding to the location image based on the location information (abbreviated as location positioning). The location information is obtained by calibrating a sample location image, and the sample location image is obtained by the image acquisition device performing image acquisition for at least one location.

[0078] In some embodiments, the goods inventory module 506 is further configured to draw a first detection block for positioning goods on the location image based on the detection result, determine a second detection block in the location image for positioning the target location, and determine the number of goods at the target location based on the overlapping degree between the first detection block and the second detection block.

[0079] The division of each module in the above goods inventory device is only used as an example for explanation. In other embodiments, the goods inventory device may be divided into different modules according to needs to realize all or part of the functions of the above goods inventory device.

[0080] All or part of each module in the above goods inventory device may be implemented by software, hardware, and combinations thereof. Each of the above modules may be embedded in a processor in a computer device in the form of hardware, or stored in a memory in a computer device in the form of software, whereby the processor can call and execute operations corresponding to each of the above modules.

[0081] In some embodiments, a computer device is provided. The computer device may be the server in FIG. 1, and its internal structure diagram is shown in FIG. 6. The computer device includes a processor, a memory, and a network interface connected to each other via a system bus. The processor of the computer device is configured to provide computing and control functions. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an execution environment for the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image information and location information. The network interface of the computer device is used to communicate with an external terminal via a network connection. The computer program is executed by the processor to execute a method for inventorying goods.

[0082] Specifically, a computer program is stored in the memory shown in FIG. 6, and when the computer program is executed by the processor, the processor executes the above method for inventorying goods. Specifically, the processor obtains a location image obtained by collecting images for at least one location for storing goods, obtains a detection result by presumptively detecting the location image, and obtains an inventory result of the goods at the location by highlighting the goods in the location image based on the detection result. Highlighting includes at least one of highlighting the recognized goods, filling in colors and text annotations, and drawing a region detection block corresponding to the goods based on the recognized goods.

[0083] Regarding estimating and detecting a location image, the processor is configured to estimate and detect the location image with an estimation and detection engine. The estimation and detection engine is obtained by serializing model parameters of a cargo detection model trained with deep learning, and the estimation and detection speed of the estimation and detection engine is faster than that of the cargo detection model.

[0084] The cargo detection model is obtained through the following training. An image collection device corresponding to any one location captures a process image during cargo transportation at any one location, and forms a training image set based on the process image. The cargo detection model is obtained by training an original detection model based on the training image set. An example of the original detection model is the yolov5 model.

[0085] In some embodiments, the location image is obtained by at least one image collection device performing image collection.

[0086] Regarding obtaining a location image, the processor is configured to obtain the location image and store the location image in an image buffer. Regarding obtaining a detection result by estimating and detecting the location image, the processor is configured to wake up a cargo estimation thread to obtain the location image from the image buffer, and call the estimation and detection engine to estimate and detect the location image, so as to obtain the detection result.

[0087] In some embodiments, the location image is obtained by an image collection thread turned on for at least one image collection device. The processor is further configured to control, via the image collection thread, at least one image collection device to stop image collection within a preset time range when the number of images stored in the image buffer reaches a preset threshold.

[0088] The cargo estimation thread is independent of the image collection thread, and an image collection thread is arranged for each of at least one image collection device.

[0089] At least one image collection device is attached above the platform, and at least one image collection device is attached at a height such that all the cargo at the location can be completely displayed within the shooting field of view of at least one image collection device.

[0090] In some embodiments, for obtaining a detection result by estimating and detecting a location image, the processor is configured to obtain a target location by detecting at least one location corresponding to the location image, and obtain the detection result by detecting the cargo at the target location with an estimation detection engine.

[0091] In some embodiments, for obtaining a target location by detecting at least one location corresponding to the location image, the processor is configured to determine an image collection device corresponding to the location image, obtain a pre-configuration file, obtain location information corresponding to the image collection device from the pre-configuration file, and obtain the target location by positioning at least one location corresponding to the location image based on the location information. The location information is obtained by calibrating a sample location image, and the sample location image is obtained by the image collection device performing image collection for at least one location.

[0092] In some embodiments, the inventory result of the goods includes the number of goods at the target location. To obtain the inventory result of the goods at the location by highlighting the goods in the location image based on the detection result, the processor draws a first detection block for positioning the goods on the location image based on the detection result, determines a second detection block in the location image for positioning the target location, and is configured to determine the number of goods at the target location based on the degree of overlap between the first detection block and the second detection block.

[0093] In some embodiments, the location image includes image information of at least one location and image information of other areas, and the other areas include non-location areas. The detection result includes whether there is a human or a forklift in the non-location area. The method for inventory of goods further includes generating a warning when there is a human or a forklift in the non-location area.

[0094] The at least one image acquisition device may be a part of the computer device or an external component of the computer device.

[0095] Those skilled in the art can understand that the structure shown in FIG. 6 is only a block diagram of some structures related to the solution of this application, and does not limit the computer device to which the solution of this application is applied. The computer device may include more or fewer components than shown in the drawings, may combine specific components, or may have different component arrangements.

[0096] In some embodiments, a computer device including a memory and a processor is further provided. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are realized.

[0097] In some embodiments, a non-volatile computer-readable storage medium storing a computer program is provided. When the computer program is executed by a processor, the steps in each of the above method embodiments are realized.

[0098] In some embodiments, a computer program product including a computer program is provided. When the computer program is executed by a processor, the steps in each of the above method embodiments are realized.

[0099] Those skilled in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by a computer program instructing relevant hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it is operable to execute the processes of the above method embodiments. Any reference to memory, database, or other media used in each embodiment of the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-transitory memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric RAM (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM), external cache memory, etc. By way of non-limiting illustrative example, RAM can have various forms, such as static random access memory (SRAM), dynamic random access memory (DRAM), etc. The database according to each embodiment of the present application can include at least one of a relational database and a non-relational database. Non-relational databases can include, but are not limited to, blockchain-based distributed databases, etc.The processor according to each embodiment of the present application can be, but is not limited to, a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc.

[0100] Each technical feature of the above embodiments can be arbitrarily combined. For the sake of brevity, not all possible combinations of each technical feature in the above embodiments are described. However, as long as there is no contradiction in the combination of those technical features, all such combinations should be considered to be within the scope of the description of this specification.

[0101] The above embodiments only show some embodiments of the present application. Although the description of the above embodiments is specific and detailed, it should not be understood as a limitation to the patent scope of the present application. Without departing from the concept of the present application, those skilled in the art can make some changes and improvements, and all of those changes and improvements should belong to the protection scope of the present application. Therefore, the protection scope of the present application should be based on the appended claims.

Claims

1. A method for inventorying goods applied to a computer, comprising: Obtaining a location image obtained by collecting an image for at least one location for storing goods; Obtaining a detection result by presumptively detecting the location image; Obtaining an inventory result of the goods at the location by highlighting the goods in the location image based on the detection result; Presumptively detecting the location image includes: Presumptively detecting the location image with a presumptive detection engine; The presumptive detection engine is obtained by serializing model parameters of a goods detection model trained by deep learning; The location image is obtained by at least one image collection device performing image collection; Obtaining the location image includes: Obtaining the location image and storing the location image in an image buffer; Obtaining the detection result by presumptively detecting the location image includes: Waking up a goods presumption thread to obtain the location image from the image buffer, and calling the presumptive detection engine to presumptively detect the location image to obtain the detection result; A method for inventorying goods, characterized by the above.

2. The presumptive detection speed of the presumptive detection engine is faster than the presumptive detection speed of the goods detection model; The method for inventorying goods according to claim 1, characterized by the above.

3. The location image is obtained by an image collection thread turned on for the at least one image collection device, and the method for inventorying goods is: Controlling, via the image collection thread, the at least one image collection device to stop image collection within a preset time range when the number of images stored in the image buffer reaches a preset threshold. The method for inventorying goods according to claim 1, characterized by the above.

4. Obtaining the detection result by estimating and detecting the location image includes: Obtaining a target location by detecting the at least one location corresponding to the location image; and Obtaining the detection result by detecting goods at the target location with the estimation and detection engine. The method for inventorying goods according to claim 2, characterized by the above.

5. Obtaining the target location by detecting the at least one location corresponding to the location image includes: Determining the image collection device corresponding to the location image; Obtaining a pre-configuration file and obtaining location information corresponding to the image collection device from the pre-configuration file; and Obtaining the target location by positioning the at least one location corresponding to the location image based on the location information, where the location information is obtained by calibrating a sample location image, and the sample location image is obtained by the image collection device collecting images for the at least one location. The method for inventorying goods according to claim 4, characterized by the above.

6. The inventory result of the goods includes the number of goods at the target location. By highlighting the goods in the location image based on the detection result, obtaining the inventory result of the goods at the location is drawing a first detection block for positioning the goods on the location image based on the detection result, determining a second detection block in the location image for positioning the target location, and determining the number of the goods at the target location based on the degree of overlap between the first detection block and the second detection block, The method for inventorying goods according to claim 4, characterized in that.

7. The location image includes image information of the at least one location and image information of other areas, and the other areas include non-location areas. The method for inventorying goods according to any one of claims 1 to 6, characterized in that.

8. The detection result includes whether a human or a forklift exists in the non-location area. The method for inventorying goods further includes generating a warning when a human or a forklift exists in the non-location area. The method for inventorying goods according to claim 7, characterized in that.

9. The highlighting includes at least one of highlighting the recognized goods, filling in colors and text annotations, and drawing a region detection block corresponding to the goods based on the recognized goods. The method for inventorying goods according to any one of claims 1 to 6, characterized in that.

10. The method for inventorying goods further includes training the goods detection model, and the training is An image acquisition device corresponding to any one of the locations captures a process image during cargo transportation at any one of the locations, and forms a training image set based on the process image. Obtaining the cargo detection model by training an original detection model based on the training image set. The method for inventorying cargo according to claim 2, characterized in that.

11. The original detection model is a yolov5 model. The method for inventorying cargo according to claim 10, characterized in that.

12. The cargo estimation thread is independent of the image acquisition thread, and the image acquisition thread is arranged for each of the at least one image acquisition device. The method for inventorying cargo according to claim 3, characterized in that.

13. The at least one image acquisition device is mounted above the platform, and the at least one image acquisition device is mounted at a height such that all the cargo at the location can be completely displayed within the shooting field of view of the at least one image acquisition device. The method for inventorying cargo according to claim 1, characterized in that.

14. The estimation detection engine Converting the format of the model file corresponding to the trained cargo detection model into the format of the file corresponding to the estimation detection engine. Generating an engine file. Serializing and storing the engine file, and calling the engine file to accelerate the estimation of the cargo detection model. The method for inventorying cargo according to claim 1, characterized in that.

15. The estimation detection engine It is obtained by converting the model file generated by the trained cargo detection model into a tensorrt engine file. The method for inventorying goods according to claim 1, characterized in that.

16. An inventory device for goods, comprising a processor and a memory, A computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the method for inventorying goods according to any one of claims 1 to 6 and 10 to 15. An inventory device for goods, characterized in that.

Images