Parking lot identification method and related device

Abstract

The invention discloses a storage location identification method and a related device, and the method comprises the steps: generating a preset storage location according to preset information, the preset information comprises the number of vertexes corresponding to a target storage location area and the number of preset storage locations, and the preset storage location comprises the preset position information of a plurality of storage locations; real-time video data of the target storage location area are obtained, the real-time video data are input into a target detection model, a storage location detection result is obtained, and the storage location detection result comprises position information of non-empty storage locations in the target storage location area; and according to the preset storage location and the storage location detection result, storage location related information is determined, the storage location related information comprises storage location state information, and the storage location state information represents that the storage location is an empty storage location or a non-empty storage location. According to the method, the training data set range of the parking space detection model can be narrowed, the model training time is shortened, and the parking space detection speed is increased while the parking space detection accuracy is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a storage location identification method and related devices. Background Art

[0002] In recent years, with the gradual development of factory automation and computer integrated manufacturing system technology, as well as the widespread application of flexible manufacturing systems and automated warehouses, smart warehousing related technologies have become the focus of academic and industrial attention. Among them, deep learning and computer vision provide many solutions and ideas for problems in this field, such as storage location detection, abnormal behavior detection, etc.

[0003] The problem with current storage location status detection is that most of them use storage location detection models for storage location detection, which requires the collection of a large number of image datasets of storage locations in two different states, empty and non-empty, to support training. In addition, the storage location area setting is easily affected by environmental factors such as lighting changes, occlusions, and image quality, resulting in inaccurate storage location detection. Therefore, the robustness of storage location detection is poor and the accuracy of the detection results is low. Summary of the invention

[0004] The embodiments of the present application provide a storage location identification method and related devices to reduce the scope of the training data set of the storage location detection model, shorten the model training time, and improve the speed and accuracy of storage location identification.

[0005] In a first aspect, an embodiment of the present application provides a storage location identification method, comprising:

[0006] Generate a preset storage location according to the preset information, wherein the preset storage location includes preset location information of multiple storage locations;

[0007] Acquire real-time video data of the target storage location area, input the real-time video data into a target detection model, and obtain a storage location detection result, wherein the storage location detection result includes location information of non-empty storage locations in the target storage location area;

[0008] The storage location related information is determined according to the preset storage location and the storage location detection result, wherein the storage location related information includes storage location status information, and the storage location status information indicates whether the storage location is an empty storage location or a non-empty storage location.

[0009] In a second aspect, an embodiment of the present application provides a storage location identification device, including:

[0010] A generating unit, configured to generate a preset storage location according to preset information, wherein the preset storage location includes preset location information of a plurality of storage locations;

[0011] an acquisition unit, configured to acquire real-time video data of the target storage location area, input the real-time video data into a target detection model, and obtain a storage location detection result, wherein the storage location detection result includes location information of the storage location where the goods exist in the target storage location area;

[0012] A determination unit is used to determine the storage location related information according to the preset storage location and the storage location detection result, wherein the storage location related information includes storage location status information, and the storage location status information indicates whether the storage location is an empty storage location or a non-empty storage location.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory storing execution instructions, wherein the memory stores one or more programs; when the processor executes the execution instructions stored in the memory, the processor executes the method described in the first aspect.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing an energy data management program, including execution instructions. When a processor of an electronic device executes the execution instructions, the processor executes the method described in the first aspect.

[0015] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute some or all of the steps described in the first aspect of the embodiment of the present application. The computer program product may be a software installation package.

[0016] It can be seen that in the embodiment of the present application, firstly, a preset storage location is generated according to preset information, wherein the preset storage location includes preset location information of multiple storage locations; real-time video data of the target storage location area is obtained, and the real-time video data is input into the target detection model to obtain a storage location detection result, wherein the storage location detection result includes the location information of non-empty storage locations in the target storage location area; storage location related information is determined according to the preset storage location and the storage location detection result, wherein the storage location related information includes storage location status information, and the storage location status information characterizes whether the storage location is an empty storage location or a non-empty storage location. By combining the detection results output by the target detection model with the storage location location information obtained according to the perspective transformation, wherein the target detection model is only obtained by training based on the training data set of non-empty storage location information, this method reduces the scope of the training data set of the storage location detection model, shortens the model training time, and improves the speed of storage location detection while ensuring the accuracy of storage location detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is a system architecture diagram of a storage location detection system provided by an embodiment of the present application;

[0019] Figure 2 It is a flow chart of a storage location identification method provided in an embodiment of the present application;

[0020] Figure 3 is a perspective transformation schematic diagram provided in an embodiment of the present application;

[0021] Figure 4 It is a flow chart of combining a test result with a preset storage location provided in an embodiment of the present application;

[0022] Figure 5 It is a structural diagram of a target detection model and a server provided in an embodiment of the present application;

[0023] Figure 6 It is a structural diagram of a storage location detection model provided in an embodiment of the present application;

[0024] Figure 7 is a structural diagram of a target detection model provided in an embodiment of the present application;

[0025] Figure 8 is a structural diagram of a backbone network module provided in an embodiment of the present application;

[0026] Fig. 9 is a schematic diagram of an angle loss provided in an embodiment of the present application;

[0027] Fig.10 It is a schematic diagram of a storage location chessboard generation interface provided in an embodiment of the present application;

[0028] Fig.11 It is a structural diagram of a scheduling system and a server provided in an embodiment of the present application;

[0029] Fig.12 This is a block diagram of the functional units of a storage location identification device provided in an embodiment of the present application;

[0030] Fig.13 It is a block diagram of the functional units of another storage location identification device provided in an embodiment of the present application;

[0031] Fig.14It is a structural schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] Currently, most warehouse location detection models are used for warehouse location detection, which requires a large amount of data support and is easily affected by environmental factors such as lighting changes, occlusions, and image quality, resulting in inaccurate warehouse location detection. Therefore, the robustness of warehouse location detection is poor and the accuracy of warehouse location detection results is low.

[0036] In view of the above problems, an embodiment of the present application provides a storage location identification method and related devices, and the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0037] See also Figure 1 , Figure 1 1 is a system architecture diagram of a storage location detection system provided by an embodiment of the present application. Figure 1As shown, the storage location detection system 100 includes a monitoring module 101, a detection module 102 and an interaction module 103. The above three modules are connected to each other in communication, wherein the monitoring module 101 includes multiple cameras and switches, each of the multiple cameras is connected to the switch, and the high-definition video stream of the camera is obtained through the switch, and then transmitted to the detection module 102 and the interaction module 103; the interaction module 103 is used to determine the relevant information of each storage location according to the overall image of the storage location area transmitted from the monitoring module 101, and after the storage location chessboard is generated, if the user is not satisfied and needs to modify it, the interaction module 103 can also be used to modify the storage location. 3 adapts and modifies the relevant information of each storage location to meet the actual production needs; the detection module 102 includes a target detection model, in which the high-definition video stream transmitted by the monitoring module 101 is input, and the location information of the storage location where the goods exist is output, and before that, the state image data set of the storage location with goods is collected by the monitoring module 101 to train the target detection model to obtain a trained model; the results output by the interaction module 103 and the detection module 102 are combined to obtain information such as the quantity information of the storage location in the storage location area, the location information of the storage location, and the storage location status information of whether there are goods.

[0038] See also Figure 2 , Figure 2 is a flow chart of a storage location identification method provided in an embodiment of the present application, such as Figure 2 As shown, the method includes the following steps.

[0039] S210, generating a preset storage location according to preset information, wherein the preset storage location includes preset location information of multiple storage locations;

[0040] Among them, the storage location refers to a specific location or area in the warehouse for storing goods. Each storage location has a unique identifier to facilitate the positioning, storage and retrieval of goods. Storage locations are usually divided according to certain rules and layouts to maximize the space utilization efficiency of the warehouse and the convenience of goods storage. Storage locations are classified and organized according to the type, size, weight, storage requirements, etc. of goods. Storage location information is usually recorded and managed, including the number, location, capacity, occupancy, etc. of the storage location, so that warehouse staff can understand the status of each storage location and perform operations such as warehousing, outbound and inventory of goods.

[0041] Among them, firstly, the overall image of the storage area is obtained through the camera, and then M vertices of the target storage area are set according to the image of the storage area, and then the preset position information of each storage location in the target storage area is determined according to the M vertices, the overall image of the storage area and the number of storage locations in the target storage area, that is, the storage location checkerboard, such as the number, size, specific location and other information of each storage location. Among them, the exemplary M is 4, and in the interface of generating the storage location checkerboard, firstly, the image of the storage area is obtained from the camera, and the 4 vertices of the storage area are determined according to the image, or the user sets the 4 vertices of the storage area according to the image, and then clicks on the 4 vertices. If there is a vertex with an incorrect labeling among the four vertices, you can click to delete the vertex, and then it will return to the previous step to reset the vertices of the storage area. After setting the 4 vertices, the user enters the number of storage locations in the storage area, and then the storage location checkerboard is generated by algorithms such as perspective change.

[0042] Among them, see Figure 3 , Figure 3 is a perspective transformation diagram provided in an embodiment of the present application. Because the camera is often installed obliquely above the storage area in reality, the objects in the picture are distorted to a certain extent, so the specific storage location is fitted through image perspective transformation, such as Figure 3 As shown, the forward image of the storage area is projected onto a new oblique plane, so as to better fit the position of the actual storage object. The transformation formula of the perspective transformation is:

[0043]

[0044] Among them, (x, y, z) are the homogeneous coordinates of the original image pixel, (X', Y', Z') are the homogeneous coordinates of the corresponding transformed image pixel, is the affine transformation parameter, which is used to define transformations such as rotation and scaling. is the translation transformation parameter,

[0045] [a 31 a 32 ] is the projection parameter, which is used to represent the edge intersection relationship after transformation. 33 =1. The above formula can be transformed into

[0046]

[0047] The above formula can be transformed into

[0048]

[0049] In the above equation, we can use 4 pairs of coordinate points, for example, the four vertices of the storage area, with the coordinate values ​​in pairs, one-to-one correspondence, to finally obtain the perspective transformation matrix between the two planes. In actual application scenarios, there may be noise or feature point mismatches, and generally more than 4 pairs of coordinate points are used to calculate the perspective transformation matrix.

[0050] If the user is not satisfied with the generated storage location grid, the generated storage location grid can be fine-tuned by manual adjustment to meet the actual production requirements. After the storage location grid is generated, the relevant information will be stored for subsequent use.

[0051] Among them, graph neural networks can also be used to generate a warehouse location checkerboard.

[0052] S220, acquiring real-time video data of the target storage location area, inputting the real-time video data into a target detection model, and obtaining a storage location detection result, wherein the storage location detection result includes location information of non-empty storage locations in the target storage location area.

[0053] The real-time video data obtained is a high-definition video stream of multiple cameras obtained through the switch. The target detection model (MA-CenterNet) that does not rely on predefined anchor points (Anchor-free) performs relevant storage location detection based on the input video stream. MA-CenterNet detects the position of the object based on the center of the detection frame and outputs key points. The model only needs to obtain the state image data set of the required storage location with goods for training, and does not need additional state image data sets of the storage location without goods, which can greatly reduce the model training time and model parameters. MA-CenterNet includes three modules, a backbone network module, a decoding module and a head module. The backbone network module is used to calculate the feature map of the storage location target image, the decoding module is used to perform feature fusion, and the head module is used to complete the output of the target detection result.

[0054] The detection result may be exemplarily outputting a picture, which includes the key center points of the storage locations where goods exist in each storage location, and a predicted storage location box outside each key center point.

[0055] S230, determining storage location related information according to the preset storage location and the storage location detection result, wherein the storage location related information includes storage location status information, and the storage location status information indicates whether the storage location is an empty storage location or a non-empty storage location.

[0056] Among them, see Figure 4 , Figure 4 is a flow chart of combining a test result with a preset storage location provided in an embodiment of the present application, such as Figure 4As shown, first, the preset storage locations can be generated through perspective transformation based on the image of the storage location area and the relevant information of the preset storage location area, that is, the number, position and quantity of each storage location in the storage location area. Among them, the image of the storage location area includes multiple boxes, each box represents a storage location, and each storage location has a number, for example, 1 represents storage location 1, 2 represents storage location 2, and the right side of storage location 1 is storage location 2. Among them, the relevant information of the preset storage location area includes the vertices of the storage location area and the number of storage locations; at the same time, the image of the storage location area is input into MA-CenterNet to determine the key center point of the storage location where the goods exist. Figure 4 The center point of the circle is the key center point, and then the detection result is output; finally, the preset storage location and the storage location detection result are combined to obtain the final storage location related information. Among them, the dotted line box indicates that there is no goods in this storage location, and the solid line box indicates that there is goods in this storage location. The size of each box indicates the size of this storage location, and the number in each box is the number of this storage location, which is convenient for quickly locating the location of the storage location.

[0057] It can be seen that in this embodiment, by combining the detection results output by MA-CenterNet with the preset storage location information obtained according to the perspective transformation, wherein the target detection model is trained only based on the training data set of non-empty storage location information, the training data set scope of the storage location detection model is narrowed, the model training time is shortened, and the speed of storage location detection is improved while ensuring the accuracy of storage location detection.

[0058] In a possible embodiment, before the real-time video data is input into the target detection model to obtain the storage location detection result, the method also includes: obtaining a storage location image data set in which goods exist in the target storage location area, and annotating the target object of each image in the storage location image data set to obtain an annotation result; constructing a backbone network to extract features from the storage location image data set to obtain a feature map; constructing a spatial pyramid pooling structure of a dilated convolution after the backbone network to perform feature fusion to obtain a fused feature map; constructing multiple prediction networks, the multiple prediction networks including a center point prediction network, a width, height and shape prediction network, an offset prediction network and an angle prediction network; based on the fused feature map and each prediction network in the multiple prediction networks, obtaining a prediction result corresponding to each prediction network; for the multiple prediction networks, based on the prediction results and the annotation results, constructing a sub-loss function corresponding to each prediction network; based on the sub-loss functions corresponding to each prediction network, constructing a total loss function, and optimizing the target detection model based on the total loss function to obtain a trained target detection model.

[0059] Among them, see Figure 5 , Figure 5 is a schematic diagram of the structure of a target detection model and a server provided in an embodiment of the present application, such as Figure 5As shown, the switch is connected to three cameras, the switch is connected to the server, and the target detection model MA-CenterNet is deployed on the server. The switch is used to transmit the high-definition video stream obtained by the camera to the server, wherein the image information of the warehouse with goods can be collected by the server, and the image information can be labeled, and then input into MA-CenterNet for a certain degree of training. According to the accuracy of the training results and the real-time requirements, MA-CenterNet is deployed on the server, so that the video stream obtained by the switch can be detected in real time. The switch can be connected to multiple cameras, and this embodiment does not limit the number of cameras.

[0060] Among them, see Figure 6 , Figure 6 is a structural diagram of a storage location detection model provided in an embodiment of the present application, such as Figure 6 As shown, the storage location detection model includes three modules, namely the backbone network module, the decoding module and the head module, wherein W×H×3 can be input as an example, which represents the width, height and number of channels of the image input to the storage location detection model, and 3 represents that the input image has three color channels of red, green and blue. For example, if the input is a color image with a width of 540 pixels and a height of 480 pixels, then the input representation may be 540*480*3. Upsampling, downsampling and deformable convolution operations are performed in the backbone network module to adjust the size and resolution of the feature map to capture the target information in the image. Among them, each state node is associated with a node in the graph and stores the state information of the node. The state of the state node can be updated over time or iterations; the aggregation node is a unit for aggregating information from neighboring nodes. It receives information from neighboring nodes and performs aggregation operations on the information. The aggregation operation can be summation, average, maximum value, minimum value, etc., depending on the aggregation function used. The decoding module finds the image features of interest and further fuses the features to obtain multi-scale fusion features. The head module includes center point prediction Heatmap, offset prediction Offset and shape prediction Width Height. The key center point is predicted by Heatmap, with a size of (W / R×H / R×C). The positions of the center points of objects of different categories are output. The output of Heatmap is refined by Offset to improve the positioning accuracy, with a size of (W / R×H / R×2). The width and height of the detection box centered on the key point are predicted by Width Height, with a size of (W / R×H / R×2), where R is the stride, which is 4 for example, C represents the target type, W represents the width, and H represents the height.

[0061] Among them, see Figure 7 , Figure 7is a structural diagram of a target detection model provided in an embodiment of the present application, such as Figure 7 As shown in Figure 2, the target detection model MA-CenterNet consists of three modules, namely the backbone network Backbone, the decoder Decoder and the head module Head, please refer to Figure 8 , Figure 8 is a structural diagram of a backbone network module provided in an embodiment of the present application, such as Figure 8 As shown in the figure, Backbone uses a lightweight convolutional neural network MobileNetV3 as the backbone network to calculate the feature map of the target image of the storage location, wherein the bottleneck layer includes multiple convolutional layers, and the exemplary MobileNetV3 small has 11 layers. The size and number of channels of the image are input into MobileNetV3, and the features are extracted through the convolutional layer, batch normalization layer (BatchNormalization, BN) and activation function hswish in sequence. Then, the input channel is converted into an expansion channel through a 1x1 convolution; the channel is expanded through a 3x3 convolution, using a step size of stride; finally, the expansion channel is converted into an output channel through a 1x1 convolution, realizing channel-separable convolution, channel attention mechanism, and residual connection. After that, the feature map is converted into the final prediction result through an average pooling layer, a fully connected layer, a batch normalization layer, an activation function layer in sequence, and finally a fully connected layer.

[0062] Among them, Figure 7 As shown, a spatial pyramid pooling module (ASPP) with dilated convolution is added to the decoder, wherein the spatial pyramid pooling module is used to further extract multi-scale information. This embodiment uses dilated convolutions with different stride rates to achieve this. ASPP mainly includes the following parts: (1) a 1×1 convolution layer and three 3×3 dilated convolutions. For the output stride output_stride=16, the rate is (6, 12, 18). If output_stride=8, the rate is doubled. The number of output channels of these convolution layers is 256, and a BN layer is included; (2) a global average pooling layer obtains the feature image related to the entire image, which is then sent to a 1×1 convolution layer and bilinearly interpolated to the original size; (3) the features of four different scales obtained from (1) and (2) are connected together in the channel dimension, and then sent to a 1×1 convolution for fusion to obtain a new feature with 256 channels.

[0063] Among them, Figure 7As shown in the figure, MA-CenterNet includes a heat map Heatmap, offset Offset, width and height Width Height and angle Angle in the head module head. The key center point is predicted by Heatmap, and the position of the center point of objects of different categories is output. The output of Heatmap is refined by Offset to improve the positioning accuracy. The width and height of the detection box centered on the key point are predicted by WidthHeight, and the angle parameters are calculated by Angle. A rotation angle information is added to each detected prediction box, so that the prediction box can more accurately describe the specific location of the warehouse, so as to realize the detection of rotated targets.

[0064] Among them, the fusion features output by the Decoder are respectively output with corresponding prediction results through Heatmap, Offset, Width Height and Angle. Based on the above prediction results and the corresponding loss function, the MA-CenterNet is optimized to obtain the trained MA-CenterNet.

[0065] It can be seen that in this embodiment, the target detection model MA-CenterNet can reduce the model training time and model parameters, and improve the speed of storage location recognition.

[0066] In a possible embodiment, the total loss function L det Satisfies the following formula: L det =L k +λ size L size +

[0067] λ off L off +λ angle L angle , where L k is the center point loss, λ size is the weight of shape loss, L size is the shape loss, λ off is the weight of the offset loss, L off is the offset loss, λ angle is the weight of the angle loss, L angle is the angle loss.

[0068] Among them, the loss function of MA-CenterNet mainly includes four parts, namely the center point loss of the heat map, the shape loss of the target length and width, the offset loss of the target center point, and the angle loss.

[0069]

[0070] Among them, x and y represent the coordinate positions in the image, c represents the category of the target in the image, α and β are two hyperparameters used to balance the difficult and easy samples, and Y xyc represents the true value, represents the predicted value, and N represents the number of key points. The predicted value of the easy-to-classify sample is close to 1. Represents a very small value, at which point the loss function L k The value of is relatively small, which plays a role in reducing the weight of the sample; when The predicted value of difficult-to-classify samples is close to 0. Indicates a larger value, then L k The value of is relatively large, so as to increase the weight of the sample; when In order to prevent the predicted value Too high is close to 1, use to serve as a penalty term, and The closer this parameter is to the midpoint, the smaller its value is, which is used to further reduce the penalty.

[0071] in, in, (w,h) is the width and height of the prediction box, (w gt ,h gt ) is the width and height of the real box, and θ is a parameter used to control the attention to shape loss, in order to avoid paying too much attention to shape loss and reducing the movement of the predicted box.

[0072] Among them, since the spatial resolution of the feature map output by the backbone network is one-fourth of the original input image, that is, each pixel on the output feature map corresponds to a 4x4 area of ​​the original image, this will bring a large error, thus introducing a bias loss value. Among them, Represents the offset value predicted by the network, p represents the coordinates of the center point of the image, and R represents the scaling factor of the Heatmap. Represents the approximate integer coordinates of the center point after scaling, and N represents the number of key points.

[0073] Among them, see Fig. 9 , Fig. 9 is a schematic diagram of an angle loss provided in an embodiment of the present application, such as Fig. 9 As shown, B is the target box, B GT is a regression frame, when B to B GT When the angle is less than α, it converges to the minimum α, otherwise it converges to β. w is the width difference between the center point of the real box and the predicted box, C h is the height difference between the center point of the real box and the predicted box, σ is the distance between the center point of the real box and the predicted box, in,

[0074] C h / σ=sinα.

[0075] It can be seen that in this embodiment, by optimizing MA-CenterNet, effective recognition of storage location targets with different resolutions can be achieved, with fast recognition speed, high accuracy, strong robustness, and rotating target detection to achieve more accurate path planning.

[0076] In a possible embodiment, the weight of the shape loss and the weight of the offset loss are the same as the weight of the angle loss.

[0077] Among them, λ size =λ off =λ anglc =0.1, which can better optimize the target detection model.

[0078] In a possible embodiment, after generating a preset storage location according to the preset information, the method further includes: if an instruction to adjust the preset location information is received, modifying the preset location information to an editable state; obtaining the adjusted preset location information, and replacing the preset location information before the adjustment with the adjusted preset location information.

[0079] Among them, see Fig.10 , Fig.10 is a schematic diagram of a storage location chessboard generation interface provided in an embodiment of the present application, such as Fig.10 As shown, click the button of automatic region generation to generate the location grid on the left side of the interface. If the location grid does not look good or the user is not satisfied, click on the location grid generation interface to manually adjust the region, and then fine-tune the generated location grid to meet user needs. After determining the final version of the location grid, the user can click the button of confirming the storage location region information to store the location related information.

[0080] It can be seen that in this embodiment, the storage location checkerboard can be fine-tuned and modified through the interactive interface.

[0081] In a possible embodiment, after generating a preset storage location according to preset information, the method further includes: receiving an information update instruction, obtaining updated preset information according to the information update instruction, the information update instruction including adding or deleting the number of vertices, or adding or deleting the number of preset storage locations; generating a preset storage location according to the updated preset information.

[0082] If an information update instruction is received, the storage location grid needs to be regenerated, for example: Fig.10As shown, if the information update instruction indicates that the vertex needs to be deleted, click the button to delete the vertex, then return to the previous step (click four vertices), reset the vertices of the storage location area, and then obtain the updated storage location information, and then, after exemplarily inputting the number of storage locations as 9, generate the storage location checkerboard again.

[0083] It can be seen that in this embodiment, the storage location grid can be updated through the interactive interface.

[0084] In a possible embodiment, the method further includes: classifying and storing the storage location related information according to parameters, where the parameters include storage location status information, storage location quantity information and / or storage location location information in the target storage location area.

[0085] Different information is stored based on different parameters, which is convenient for the subsequent dispatching system to call the corresponding interface for use and display it on the operation interface. Fig.11 , Fig.11 is a schematic diagram of the structure of a scheduling system and a server provided in an embodiment of the present application, such as Fig.11As shown, the switch is connected to three cameras, the switch is connected to the server, MA-CenterNet is deployed on the server, and the scheduling system establishes a connection with the server so as to be able to read the real-time related information of the storage location stored on the server, which can be achieved through a network connection or a direct physical connection; after the connection, the scheduling system can read the related information about the storage location and inventory from the server, which may include the location of the storage location, inventory quantity, type of goods, status of goods, etc., for example, the number of storage locations with goods: 4, the number of free storage locations: 2, and more specifically, the coordinate information of the storage location, such as No. 1 has goods, the specific coordinates of No. 1 are (110, 234), No. 2 has goods, the specific coordinates of No. 2 are (324, 234), No. 4 has goods, the specific coordinates of No. 4 are (110, 468), No. 6 has goods, No. 6 has goods, The specific coordinates of No. (538, 468), No. 3 and No. 5 are idle, etc., where the number in the information refers to the warehouse number, such as No. 1 refers to warehouse 1; then the dispatching system analyzes the information read and combines it with the demand, such as which goods need to be replenished, which warehouses need to be used, and how to optimally arrange the path of the automatic guided vehicle, etc., and then according to the results of the demand analysis, the dispatching system generates corresponding dispatching instructions, such as which warehouse the automatic guided vehicle goes to to pick up goods, which warehouse to put goods, whether it needs to be transported, etc.; finally, the dispatching system sends the generated instructions to the corresponding automatic guided vehicle, which can be achieved through wireless network, Bluetooth or other communication methods; after receiving the instructions, the automatic guided vehicle will perform corresponding operations according to the requirements of the instructions, such as automatically navigating to the specified warehouse, picking up goods, putting goods, etc. In the process of the automatic guided vehicle performing the operation, the dispatching system will continuously monitor its status and position, and make corresponding adjustments based on the feedback information, such as re-planning the path of the automatic guided vehicle, adjusting the task priority, etc., and when the automatic guided vehicle completes the specified operation, the dispatching system will record the result of the task and prepare for the next task.

[0086] It can be seen that in this embodiment, the dispatching system can read the real-time relevant information of the storage location stored on the server according to the demand and dispatch the automatic guided vehicle operation.

[0087] For the above embodiments, please refer to Fig.12 , Fig.12 is a block diagram of the functional units of a storage location identification device provided in an embodiment of the present application, such as Fig.12As shown, the storage location identification device 1200 includes: a generation unit 1210, which is used to generate a preset storage location according to preset information, wherein the preset storage location includes preset location information of multiple storage locations; an acquisition unit 1220, which is used to acquire real-time video data of the target storage location area, input the real-time video data into a target detection model, and obtain a storage location detection result, wherein the storage location detection result includes location information of storage locations where goods exist in the target storage location area; a determination unit 1230, which is used to determine storage location related information according to the preset storage location and the storage location detection result, wherein the storage location related information includes storage location status information, and the storage location status information indicates whether the storage location is an empty storage location or a non-empty storage location.

[0088] In a possible embodiment, in terms of generating preset storage locations based on preset information, the generation unit 1210 is specifically used to: obtain the number of vertices and the number of preset storage locations corresponding to the target storage location area; determine the preset storage location based on the number of vertices and the number of preset storage locations; wherein the storage location related information also includes storage location quantity information and / or storage location location information.

[0089] In a possible embodiment, before inputting the real-time video data into the target detection model to obtain the storage location detection result, the storage location identification device 1200 is also specifically used to: obtain a storage location image data set in which goods exist in the target storage location area, and mark the target object of each image in the storage location image data set to obtain a marking result; construct a backbone network to extract features from the storage location image data set to obtain a feature map; construct a spatial pyramid pooling structure of void convolution after the backbone network to perform feature fusion to obtain a fused feature map; construct multiple prediction networks, the multiple prediction networks include a center point prediction network, a width and height prediction network, an offset prediction network and an angle prediction network; obtain a target prediction result based on the fused feature map and each prediction network in the multiple prediction networks; train a model according to the target prediction result to obtain a trained target detection model.

[0090] In a possible embodiment, after obtaining a storage location image data set in which goods exist in the target storage location area, the storage location identification device 1200 is further specifically used to: label the target object in each image in the storage location image data set to obtain a labeling result; train the model according to the prediction result to obtain a trained target detection model, including: for the multiple prediction networks, based on the target prediction results and the labeling results, constructing a sub-loss function corresponding to each prediction network; based on the sub-loss function corresponding to each prediction network, constructing a total loss function, and optimizing the target detection model based on the total loss function to obtain a trained target detection model.

[0091] In a possible embodiment, in terms of obtaining a target prediction result based on the fused feature map and each prediction network in the multiple prediction networks, the storage location identification device 1200 is also specifically used to: determine the center point coordinates of the target object according to the fused feature map; determine a position offset based on the fused feature map and the center point coordinates, and determine new center point coordinates according to the position offset; determine a width, height and rotation angle of a detection frame centered on the new center point coordinates according to the fused feature map and the new center point coordinates; obtain a target prediction result according to the rotation angle, the position offset, the center point coordinates and the width and height of the detection frame, the target prediction result including a predicted center point and a predicted detection frame with rotation.

[0092] In a possible embodiment, in terms of calculating the total loss function, the storage location identification device 1200 is further specifically used for: the total loss function L det Satisfies the following formula: L det =L k +λ size L size +λ off L off +λ angle L angle , where L k is the center point loss, λ size is the weight of shape loss, L size is the shape loss, λ off is the weight of the offset loss, L off is the offset loss, λ angle is the weight of the angle loss, L angle is the angle loss.

[0093] In a possible embodiment, in terms of calculating angle loss, the storage location identification device 1200 is also specifically used to: determine the center point of the real detection frame based on the labeling result; obtain the center point of the predicted detection frame in the target detection result; and determine the angle loss based on the center point of the real detection frame and the center point of the predicted detection frame.

[0094] In a possible embodiment, in terms of calculating the angle loss, the storage location identification device 1200 is further specifically used for: the angle loss L angle Satisfies the following formula: Among them, C h is the height difference between the center point of the real detection frame and the center point of the predicted detection frame, and σ is the distance between the center point of the real detection frame and the center point of the predicted detection frame.

[0095] In a possible embodiment, in terms of calculating shape loss, the storage location identification device 1200 is also specifically used to: determine the width and height of the real detection frame according to the labeling result; obtain the width and height of the predicted detection frame in the target detection result; and determine the shape loss according to the width and height of the real detection frame and the width and height of the predicted detection frame.

[0096] In a possible embodiment, in terms of calculating the shape loss, the storage location identification device 1200 is further specifically used for: the shape loss L size Satisfies the following formula: in, w is the width of the predicted detection box, h is the height of the predicted detection box, w gt is the width of the real detection box, h gt is the height of the true detection box, and θ is a parameter used to control the focus on shape loss.

[0097] In a possible embodiment, in terms of calculating the total loss function, the storage location identification device 1200 is further specifically configured to: the weight of the shape loss and the weight of the offset loss are both the same as the weight of the angle loss.

[0098] In a possible embodiment, after generating a preset storage location according to preset information, the storage location identification device 1200 is further specifically used to: if an instruction to adjust the preset position information is received, modify the preset position information to an editable state; obtain the adjusted preset position information, and replace the preset position information before adjustment with the adjusted preset position information.

[0099] In a possible embodiment, after generating a preset storage location according to preset information, the storage location identification device 1200 is also specifically used to: receive an information update instruction, obtain updated preset information according to the information update instruction, and the information update instruction includes adding or deleting the number of vertices, or adding or deleting the number of preset storage locations; generate a preset storage location according to the updated preset information.

[0100] In a possible embodiment, in terms of storage location related information, the storage location identification device 1200 is also specifically used to: classify and store the storage location related information according to parameters, and the parameters include storage location status information, storage location quantity information and / or storage location location information in the target storage location area.

[0101] In a possible embodiment, in terms of determining the preset storage location according to the number of vertices and the number of preset storage locations, the generation unit 1210 is specifically used to: generate a forward image of the target storage location area according to the number of vertices and the number of preset storage locations, the forward image including reference position information of each storage location in a plurality of storage locations; project the forward image into an oblique image through perspective transformation; and determine the oblique image as the preset storage location.

[0102] It can be understood that since the method embodiment and the device embodiment are different presentation forms of the same technical concept, the content of the method embodiment part in this application should be synchronously adapted to the device embodiment part and will not be repeated here.

[0103] In the case of integrated units, see Fig.13 , Fig.13 1 is a block diagram of the functional units of another storage location identification device provided in an embodiment of the present application. Fig.13 As shown, the storage location identification device 1200 includes: a processing module 1202 and a communication module 1201. The processing module 1202 is used to control and manage the actions of the storage location identification device 1200, for example, to execute the steps of the generation unit 1210, the acquisition unit 1220 and the determination unit 1230, and / or to execute other processes of the technology described herein. The communication module 1201 is used for the interaction between the storage location identification device 1200 and other devices. Fig.13 As shown, the storage location identification device 1200 may further include a storage module 1203, and the storage module 1203 is used to store program codes and data of the storage location identification device.

[0104] Among them, the processing module 1202 can be a processor or a controller, for example, a central processing unit (CPU), a general processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements a computing function, for example, a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 1201 can be a transceiver, an RF circuit or a communication interface, and the like. The storage module 1203 can be a memory.

[0105] All relevant contents of each scenario involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here. Figure 2 The storage location identification method shown.

[0106] See also Fig.14 , Fig.14 is a schematic diagram of the structure of an electronic device proposed in an embodiment of the present application, such as Fig.14 As shown, the electronic device 1400 includes a processor 1410, a memory 1420, a communication interface 1430 and one or more programs 1421. The one or more programs 1421 are stored in the memory and are configured to be executed by the processor. When the program is executed, it includes part or all of the steps of any one of the cruising range estimation methods recorded in the above method embodiments. The processor, memory and communication interface are interconnected and complete communication with each other.

[0107] The memory may be a volatile memory such as a dynamic random access memory DRAM, or a non-volatile memory such as a mechanical hard disk. The memory is used to store a set of executable program codes, and the processor is used to call the executable program codes stored in the memory, and may execute some or all steps of any energy data management method recorded in the above-mentioned embodiment of the range estimation method.

[0108] It can be seen that the electronic device 1400 described in the embodiment of the present application first generates a preset storage location according to preset information, wherein the preset information includes the number of vertices and the number of preset storage locations corresponding to the target storage location area, and the preset storage location includes preset position information of multiple storage locations; obtains real-time video data of the target storage location area, inputs the real-time video data into the target detection model, obtains a storage location detection result, and the storage location detection result includes the location information of the storage location where goods exist in the target storage location area; determines storage location related information according to the preset storage location and the storage location detection result, and the storage location related information includes the number information of the storage location in the target storage location area, the location information of the storage location, and the storage location status information whether there are goods. By combining the detection result output by the target detection model with the preset storage location location information obtained according to the perspective transformation, the speed of storage location detection is improved while ensuring the accuracy of storage location detection.

[0109] An embodiment of the present application also provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute part or all of the steps of any method described in the above method embodiments, and the above computer includes an electronic device.

[0110] The embodiment of the present application also provides a computer program product, the computer program product includes a non-transitory computer-readable storage medium storing a computer program, the computer program is operable to cause a computer to execute some or all of the steps of any method described in the method embodiment. The computer program product may be a software installation package, and the computer includes an electronic device.

[0111] It should be noted that, for the sake of simplicity, the aforementioned method implementations are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the implementations described in the specification are all optional implementations, and the actions and modules involved are not necessarily required by the present application.

[0112] In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed device can be implemented in other ways. For example, the device implementation described above is only schematic, such as the division of the units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0115] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software program module.

[0116] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0117] A person skilled in the art may understand that all or part of the steps in the various methods of the above-mentioned embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable memory, and the memory may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0118] The above is a detailed introduction to the implementation methods of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above implementation methods is only used to help understand the method and core idea of ​​the present application. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A storage location identification method, characterized in that: include: Generate a preset storage location according to the preset information, wherein the preset storage location includes preset location information of multiple storage locations; Acquire real-time video data of a target storage location area, input the real-time video data into a target detection model, and obtain a storage location detection result, wherein the storage location detection result includes location information of non-empty storage locations in the target storage location area; The storage location related information is determined according to the preset storage location and the storage location detection result, wherein the storage location related information includes storage location status information, and the storage location status information indicates whether the storage location is an empty storage location or a non-empty storage location.

2. The method according to claim 1, characterized in that Generating a preset storage location according to preset information includes: Obtain the number of vertices and the number of preset storage locations corresponding to the target storage location area; Determine the preset storage location according to the number of vertices and the number of preset storage locations; The storage location related information also includes storage location quantity information and / or storage location location information.

3. The method according to claim 1, characterized in that Before inputting the real-time video data into the target detection model to obtain the storage location detection result, the method further includes: Acquire a storage location image dataset of goods in the target storage location area; Constructing a backbone network to extract features of the storage location image data set to obtain a feature map; After the backbone network, a spatial pyramid pooling structure of dilated convolution is constructed to fuse features and obtain a fused feature map. Constructing multiple prediction networks, wherein the multiple prediction networks include a center point prediction network, a width and height prediction network, an offset prediction network, and an angle prediction network; Obtaining a target prediction result based on the fused feature map and each prediction network in the multiple prediction networks; A model is trained according to the target prediction results to obtain a trained target detection model.

4. The method according to claim 3, characterized in that After obtaining the storage location image dataset of goods in the target storage location area, the method further includes: Annotating the target object of each image in the storage location image dataset to obtain an annotation result; The step of training a model according to the prediction result to obtain a trained target detection model includes: For the multiple prediction networks, based on the target prediction result and the labeling result, construct a sub-loss function corresponding to each prediction network; Based on the sub-loss functions corresponding to each prediction network, a total loss function is constructed, and the target detection model is optimized based on the total loss function to obtain a trained target detection model.

5. The method according to claim 4, characterized in that The obtaining a target prediction result based on the fused feature map and each prediction network in the multiple prediction networks includes: Determine the center point coordinates of the target object according to the fused feature map; Determining a position offset based on the fused feature map and the center point coordinates, and determining new center point coordinates according to the position offset; Determine the width, height and rotation angle of the detection frame centered on the new center point coordinates according to the fused feature map and the new center point coordinates; According to the rotation angle, the position offset, the center point coordinates and the width and height of the detection frame, a target prediction result is obtained, where the target prediction result includes a predicted center point and a predicted detection frame with rotation.

6. The method according to claim 4, characterized in that The total loss function L det Satisfies the following formula: L det =L k +λ size L size +λ off L off +λ angle L angle , Among them, L k is the center point loss, λ size is the weight of shape loss, L size is the shape loss, λ off is the weight of the offset loss, L off is the offset loss, λ angle is the weight of the angle loss, L angle is the angle loss.

7. The method according to claim 6, characterized in that The method further comprises: Determine the center point of the real detection frame according to the labeling result; Get the center point of the predicted detection box in the target detection result; An angle loss is determined according to a center point of the real detection frame and a center point of the predicted detection frame.

8. The method according to claim 7, characterized in that The angle loss L angle Satisfies the following formula: Among them, C h is the height difference between the center point of the real detection frame and the center point of the predicted detection frame, and σ is the distance between the center point of the real detection frame and the center point of the predicted detection frame.

9. The method according to claim 6, characterized in that The method further comprises: Determine the width and height of the real detection frame according to the labeling result; Get the width and height of the predicted detection box in the target detection result; The shape loss is determined according to the width and height of the real detection box and the width and height of the predicted detection box.

10. The method according to claim 9, characterized in that The shape loss L size Satisfies the following formula: in, w is the width of the predicted detection box, h is the height of the predicted detection box, w gt is the width of the real detection box, h gt is the height of the true detection box, and θ is a parameter used to control the focus on shape loss.

11. The method according to claim 6, characterized in that The weight of the shape loss and the weight of the offset loss are both the same as the weight of the angle loss.

12. The method according to claim 2, characterized in that: After generating the preset storage location according to the preset information, the method further includes: If an instruction to adjust the preset position information is received, modifying the preset position information to an editable state; The adjusted preset position information is obtained, and the adjusted preset position information is used to replace the preset position information before the adjustment.

13. The method according to claim 2 or 12, characterized in that: After generating the preset storage location according to the preset information, the method further includes: Receive an information update instruction, and obtain updated preset information according to the information update instruction, wherein the information update instruction includes adding or deleting the number of vertices, or adding or deleting the number of preset storage locations; A preset storage location is generated according to the updated preset information.

14. The method according to claim 2, characterized in that The method further comprises: The storage location related information is classified and stored according to parameters, wherein the parameters include storage location status information, storage location quantity information and / or storage location position information in the target storage location area.

15. The method according to claim 2, characterized in that Determining the preset storage location according to the number of vertices and the number of preset storage locations includes: Generating a forward image of the target storage location area according to the number of vertices and the number of preset storage locations, wherein the forward image includes reference position information of each storage location in the plurality of storage locations; Projecting the forward image into an oblique image through perspective transformation; The oblique view image is determined as the preset storage location.

16. A storage location identification device, characterized in that: include: A generating unit, configured to generate a preset storage location according to preset information, wherein the preset storage location includes preset location information of a plurality of storage locations; an acquisition unit, configured to acquire real-time video data of the target storage location area, input the real-time video data into a target detection model, and obtain a storage location detection result, wherein the storage location detection result includes location information of the storage location where the goods exist in the target storage location area; A determination unit is used to determine the storage location related information according to the preset storage location and the storage location detection result, wherein the storage location related information includes storage location status information, and the storage location status information indicates whether the storage location is an empty storage location or a non-empty storage location.

17. An electronic device, characterized in that: The method comprises a processor and a memory storing execution instructions, wherein the memory stores one or more programs; when the processor executes the execution instructions stored in the memory, the processor executes the method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that: An energy data management program is stored, including execution instructions. When a processor of an electronic device executes the execution instructions, the processor executes the method according to any one of claims 1-15.

Images