Method, device, computer equipment and storage medium for determining box fork slot area

Abstract

The present application relates to the field of computer vision technology, and provides a method, device, computer equipment and storage medium for determining a fork groove area of ​​a box. The present application can efficiently and accurately locate the fork groove area in the box image, and is also convenient for comprehensive and effective monitoring of whether the box fork groove meets industry standards. The method includes: acquiring a box image, and then using a pre-built fork groove edge detection network to determine multiple fork groove candidate areas in the box image, and then combining these fork groove candidate areas to obtain multiple fork groove candidate area groups, and finally, based on the size constraints and layout constraints formed on the fork groove candidate area groups based on the standard size of the box's fork groove and the standard layout of the fork groove, the fork groove area in the box image can be determined from these fork groove candidate area groups.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, computer equipment and storage medium for determining a fork slot area of ​​a box. Background Art

[0002] With the rapid development of logistics technology, containers and other boxes with certain strength, rigidity and specifications are usually used for loading, storage and transportation of goods. The fork groove is a component of the side panel of a box such as a container, which is generally present on a 20-foot container to facilitate the empty container to be carried by a forklift. In actual application scenarios, if certain parts on the surface of the box are not manufactured according to specifications, it will have a great impact on loading, storage and transportation. For example, when a forklift is carrying a container in a container yard, if the fork groove of the container does not meet the specifications, it is likely to cause significant damage to both the container and the forklift.

[0003] In current technology, the position of the fork slot of the box is generally measured manually, which takes a lot of time and has low efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment and storage medium for determining the fork groove area of ​​a box body in order to solve the above technical problems.

[0005] A method for determining a box fork groove area, the method comprising:

[0006] Get the box image;

[0007] Using a pre-built fork groove edge detection network, a plurality of fork groove candidate regions in the box image are determined; the regional edges of the fork groove candidate regions are determined according to the detection result of the fork groove edge in the box image by the fork groove edge detection network;

[0008] Combining the plurality of fork groove candidate regions to obtain a fork groove candidate region group;

[0009] Based on the size constraints and layout constraints formed on the fork slot candidate area group by the fork slot standard size and the fork slot standard layout of the box, the fork slot area in the box image is determined from the fork slot candidate area group.

[0010] In one embodiment, the method comprises:

[0011] The relative sizes of the first distance and the second distance corresponding to each fork groove candidate area group are obtained to determine whether the box image has image distortion.

[0012] A device for determining a box fork groove area, the device comprising:

[0013] An image acquisition module, used to acquire a box image;

[0014] A candidate determination module is used to determine a plurality of fork groove candidate regions in the box image by using a pre-built fork groove edge detection network; the region edges of the fork groove candidate regions are determined according to the detection result of the fork groove edge in the box image by the fork groove edge detection network;

[0015] A region combination module, used for combining the plurality of fork groove candidate regions to obtain a fork groove candidate region group;

[0016] The area determination module is used to determine the fork groove area in the box image from the fork groove candidate area group based on the size constraints and layout constraints formed on the fork groove candidate area group by the fork groove standard size and fork groove standard layout of the box.

[0017] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0018] Acquire a box image; determine a plurality of fork groove candidate areas in the box image by using a pre-constructed fork groove edge detection network; determine the area edge of the fork groove candidate area according to the detection result of the fork groove edge in the box image by the fork groove edge detection network; combine the plurality of fork groove candidate areas to obtain a fork groove candidate area group; determine the fork groove area in the box image from the fork groove candidate area group based on the size constraints and layout constraints formed on the fork groove candidate area group by the standard size of the fork groove and the standard layout of the fork groove.

[0019] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0020] Acquire a box image; determine a plurality of fork groove candidate areas in the box image by using a pre-constructed fork groove edge detection network; determine the area edge of the fork groove candidate area according to the detection result of the fork groove edge in the box image by the fork groove edge detection network; combine the plurality of fork groove candidate areas to obtain a fork groove candidate area group; determine the fork groove area in the box image from the fork groove candidate area group based on the size constraints and layout constraints formed on the fork groove candidate area group by the standard size of the fork groove and the standard layout of the fork groove.

[0021] The above-mentioned method, device, computer equipment and storage medium for determining the fork groove area of ​​the box body obtains the box body image, and then the pre-built fork groove edge detection network can be used to determine multiple fork groove candidate areas in the box body image, and then these fork groove candidate areas are combined to obtain multiple fork groove candidate area groups, and finally the fork groove area in the box body image can be determined from these fork groove candidate area groups based on the size constraints and layout constraints formed on the fork groove candidate area groups by the standard size and layout of the fork groove of the box body. The scheme can be combined with computer vision technology, firstly the fork groove area is roughly located by using the fork groove edge detection network, and then the roughly located candidate areas are grouped to form multiple fork groove candidate area groups, and finally the fork groove area is accurately determined from the candidate area group by the size constraints and layout constraints formed on the fork groove candidate area groups by the standard size and layout of the fork groove, so as to realize efficient and accurate positioning of the fork groove area in the box body image, and the scheme can be implemented based on computer equipment to facilitate comprehensive and effective monitoring of whether the fork grooves of the yard set and the box body just produced and shipped meet the industry specifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A diagram showing an application environment of a method for determining a box fork slot area in an embodiment;

[0023] Figure 2 A schematic diagram of a process of determining a box fork slot area in one embodiment;

[0024] FIG3( a ) is a schematic diagram of a box image in one embodiment;

[0025] FIG3( b ) is a schematic diagram of the positioning result of the fork groove area in one embodiment;

[0026] Figure 4 is a schematic diagram of a fork groove candidate area in one embodiment;

[0027] Figure 5 A schematic diagram of a process for determining a candidate fork groove area in one embodiment;

[0028] Figure 6 A schematic diagram of the detection result of the fork groove edge in one embodiment;

[0029] Figure 7 A schematic diagram of a straight line detection result in an embodiment;

[0030] Figure 8 is a schematic diagram of a merging result of two types of candidate edge lines in one embodiment;

[0031] Fig. 9 A schematic flow chart of the steps of obtaining the detection result of the fork slot edge in the box image by the fork slot edge detection network in one embodiment;

[0032] Fig.10 is a schematic diagram of a box image structure in one embodiment;

[0033] Fig.11 is a schematic diagram of a partial image of a box in one embodiment;

[0034] Fig.12 This is the original network structure diagram of the HRNet network;

[0035] Fig.13 Schematic diagram of the principle of applying HRNet network to detection or segmentation;

[0036] Fig.14 It is a schematic diagram of the principle of outputting detection results of a fork groove edge detection network in one embodiment;

[0037] Fig.15 A schematic diagram of a process for constructing a fork slot edge detection network in one embodiment;

[0038] FIG16( a ) is a schematic diagram showing the result of locating the fork groove area in a box image in an application example;

[0039] FIG16( b ) is a schematic diagram showing the result of locating the fork groove area in the box image in another application example;

[0040] FIG16( c ) is a schematic diagram showing the result of locating the fork groove area in the box image in another application example;

[0041] FIG. 17( a ) is a schematic diagram of the original business process for positioning the fork slot area of ​​a container;

[0042] FIG17( b ) is a schematic diagram of a business process for locating a fork slot area of ​​a container in one embodiment;

[0043] Fig.18 It is a structural block diagram of a device for determining a box fork groove area in one embodiment;

[0044] Fig.19 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] Some abbreviations and terms used in this application are defined as follows:

[0047] Linear scanning imaging: Linear scanning imaging means that when an object is imaged, a line image or even a two-dimensional image is formed each time a line is scanned on the image plane, and finally a complete image is achieved by splicing along the direction of the object's movement. It is referred to as line scanning imaging.

[0048] 2D image: A 2D image is also called a plane image. A 2D image only has X-axis and Y-axis.

[0049] Hough transform: used to identify features in objects, such as line features, by quantizing the Hough parameter space into finite intervals or accumulator units to extract straight line information.

[0050] HRNet: A widely used deep learning-based network that can maintain high-resolution representation throughout the entire process, hence the name High-Resoultion Net.

[0051] Fork pocket: A part of the side panel of the container that facilitates the empty container to be transported by forklift. Taking the container as an example, fork pockets are generally found on 20-foot containers.

[0052] In the positioning monitoring related to the box, depth information is generally required, but the fork slot of the box is a hole, and its depth information is not controllable. The present application can locate the fork slot area of ​​the box without using the depth map or 3D (3D) point cloud of the box. Among them, the automatic positioning method of the container fork slot area provided by the present application can be combined with computer vision technology, and the 2D image of the box is used to efficiently and accurately locate the fork slot area on the image.

[0053] The method for determining the box fork groove area provided in this application can be applied to Figure 1 In the application environment shown, in the application environment, the terminal 110 communicates with the server 120 through the network. Among them, the terminal 110 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 110 and the server 120 can be directly or indirectly connected via wired or wireless communication, and this application is not limited here.

[0054] Specifically, the method for determining the box fork slot area provided in the present application can be executed by the terminal 110 or by the server 120, or by the terminal 110 and the server 120. As an example, the following is described by taking the execution by the terminal 110 and the execution by the terminal 110 and the server 120 as examples.

[0055] In an exemplary embodiment, the terminal 110 can obtain a box image, use a pre-built fork groove edge detection network to determine multiple fork groove candidate areas in the box image, and then combine the multiple fork groove candidate areas to obtain a fork groove candidate area group, and finally determine the fork groove area in the box image from the fork groove candidate area group based on the size constraints and layout constraints formed on the fork groove candidate area group by the standard size and standard layout of the fork groove of the box. This exemplary technical solution can be executed by the terminal 110 located at the box transportation site alone, without the help of the server 120, that is, the terminal 110 can complete the accurate and efficient positioning of the box fork groove area without connecting to the Internet, is not easily affected by the stability of the Internet, and has high reliability.

[0056] In another exemplary embodiment, the terminal 110 may obtain a box image, and then the terminal 110 may send the box image to the server 120. After receiving the box image, the server 120 may use a pre-built fork groove edge detection network to determine multiple fork groove candidate areas in the box image, and then the server 120 combines the multiple fork groove candidate areas to obtain a fork groove candidate area group. Then, the server 120 determines the fork groove area in the box image from the fork groove candidate area group based on the size constraints and layout constraints formed on the fork groove candidate area group by the standard size and standard layout of the fork groove of the box, and finally the server 120 feeds back the positioning result to the terminal 110. This exemplary technical solution can be executed by the terminal 110 located at the box transportation site and the remote server 120. This solution can reduce the data processing pressure of the terminal 110, and this solution provides a fork groove area positioning service by the server 120, that is, the server 120 can serve as a provider of artificial intelligence cloud service AIaaS (AI as aService, Chinese for "AI as a service"). Among them, the artificial intelligence cloud service can split a variety of AI services and provide independent or packaged services in the cloud. Specifically in this embodiment, the server 120 provides the terminal 110 with an AI service for locating the fork slot area based on the fork slot edge detection network. The terminal 110 can access and use the artificial intelligence service provided by the server 120 through an API interface. The terminal 110 can also use the server 120 to deploy and operate a dedicated cloud artificial intelligence service, so as to implement comprehensive and effective monitoring of whether the fork slots of the container just produced and shipped from the yard meet the industry specifications.

[0057] In one embodiment, Figure 2 As shown, a method for determining the fork groove area of ​​a box body is provided, and the method is applied to Figure 1 The terminal 110 in the example is used as an example to illustrate, and the following steps are included:

[0058] Step S201, obtaining a box image;

[0059] Among them, the box image refers to an image of a box that needs to be positioned in the fork groove area. The image can be a 2D image of the box captured by a line scan camera. Exemplarily, as shown in FIG3(a), it is a line scan 2D grayscale image of a container side panel. In order to facilitate an intuitive understanding of the effect achieved by the method for determining the box fork groove area provided in the present application, FIG3(b) shows the positioning result of the fork groove area on the line scan 2D grayscale image of the container side panel corresponding to FIG3(a), and the positioning result includes a first fork groove area 301 and a second fork groove area 302.

[0060] Step S202, using a pre-built fork groove edge detection network, determining a plurality of fork groove candidate regions in the box image.

[0061] This step mainly involves the terminal 110 using a pre-built fork groove edge detection network to detect the fork groove candidate area in the box image, that is, the terminal 110 uses the fork groove edge detection network to preliminarily locate the fork groove area in the box image, and roughly obtains a plurality of suspected fork groove areas on the box image, and these suspected fork groove areas are used as fork groove candidate areas. Among them, the fork groove edge detection network refers to a neural network that has the ability to recognize, detect or segment the edge of the fork groove area in the image. In a specific implementation, when the terminal 110 uses the fork groove edge detection network to detect multiple fork groove candidate areas in the box image, the terminal 110 can input an entire box image into the fork groove edge detection network for detection, or divide the entire box image into multiple images and input them into the fork groove edge detection network for detection, or only input part of the box image into the fork groove edge detection network for detection. After the image is input into the fork groove edge detection network, the fork groove edge detection network outputs the detection result of the fork groove edge in the box image, and the detection result includes marking the position of the fork groove in the image. The general marking form can be marking the edge pixel points of the fork groove area, so that the terminal 110 can determine the area edge of the fork groove candidate area in the box image according to these marks.

[0062] Step S203, combining multiple fork groove candidate regions to obtain a fork groove candidate region group;

[0063] In this step, the terminal 110 combines the fork groove candidate areas to obtain multiple fork groove candidate area groups. In the actual scenario of identifying and locating the fork groove of the box, there are usually areas on the box that are very similar to the fork groove, which interfere with the identification and positioning of the real fork groove area, such as Figure 4 As shown, Figure 4 The fork groove candidate region 411 identified in the portion 410 is not a true fork groove, but Figure 4 The other fork groove candidate area 421 marked in the part 420 of the fork groove is the real fork groove. In this case, it is difficult to distinguish even with the naked eye, so it is necessary to further screen the multiple fork groove candidate areas obtained based on the fork groove edge detection network. There are generally two fork groove areas on the box body, so in this step, the multiple fork groove candidate areas obtained in step S202 can be combined in pairs to obtain multiple fork groove candidate area groups or fork groove candidate area pairs.

[0064] Step S204, based on the size constraints and layout constraints of the fork slot candidate area group formed by the fork slot standard size and the fork slot standard layout of the box, the fork slot area in the box image is determined from the fork slot candidate area group.

[0065] In this step, the size of the fork groove of the box and the position of the fork groove on the box correspond to industry standards, so each box has a standard fork groove size and a standard fork groove layout. The standard fork groove size refers to the size parameters of the fork groove on the box in accordance with the industry standard, which may include length and width, etc.; the standard fork groove layout refers to the position of the fork groove on the box in accordance with the industry standard, which may include the relative relationship with the edge position of the box, etc. The standard fork groove size and the standard fork groove layout can form size constraints and layout constraints for the fork groove candidate area group to be identified and located. The size constraint refers to the constraint on the size of the fork groove included in the fork groove candidate area group, such as whether the size of each fork groove in the fork groove candidate area group is consistent, etc.; the layout constraint refers to the constraint on the layout position of the fork groove included in the fork groove candidate area group, such as the relative position relationship of each fork groove in the fork groove candidate area group, etc. Therefore, after grouping the multiple fork groove candidate areas in pairs to obtain multiple fork groove candidate area groups through step S203, the terminal 110 can use the size constraints and layout constraints formed by the standard size of the fork groove and the standard layout of the fork groove of the box on the fork groove candidate area group, and finally determine the fork groove area in the box image from the fork groove candidate area group. Specifically, the terminal 110 can select a fork groove candidate area group that meets the aforementioned size constraints and layout constraints from the fork groove candidate area group as the fork groove final selection area, and finally determine the fork groove area included in the fork groove final selection area as the fork groove area in the box image, thereby realizing the recognition and positioning of the fork groove. The example of the final positioning result of the fork groove can be referred to Figure 3(b).

[0066] In the above-mentioned method for determining the fork groove area of ​​a box, the terminal 110 obtains a box image, and then the terminal 110 can use a pre-built fork groove edge detection network to determine multiple fork groove candidate areas in the box image, and then the terminal 110 combines these fork groove candidate areas to obtain multiple fork groove candidate area groups, and finally the terminal 110 can determine the fork groove area in the box image from these fork groove candidate area groups based on the size constraints and layout constraints formed on the fork groove candidate area groups by the standard size and standard layout of the fork groove of the box. This solution can be combined with computer vision technology, firstly use the fork groove edge detection network to roughly locate the fork groove area, then group the roughly located candidate areas to form multiple fork groove candidate area groups, and finally accurately determine the fork groove area from the candidate area group by the size constraints and layout constraints formed on the fork groove candidate area groups by the standard size and standard layout of the fork groove, so as to realize efficient and accurate positioning of the fork groove area in the box image, and this solution can be implemented based on computer equipment to facilitate comprehensive and effective monitoring of whether the fork grooves of the yard set and the box forks just produced and shipped meet the industry specifications.

[0067] Combination Figure 5 The fork groove edge detection network constructed in advance in the aforementioned step S202 is used to determine a plurality of fork groove candidate regions in the box image. Figure 5 As shown, step S202 may specifically include:

[0068] Step S501, obtaining the detection result of the fork groove edge in the box image by the fork groove edge detection network;

[0069] In this step, the terminal 110 may input the box image into the fork groove edge detection network, so that the fork groove edge detection network outputs the detection result of the fork groove edge in the box image. In some embodiments, the terminal 110 may also segment the box image and then input it into the fork groove edge detection network to detect the fork groove edge, obtain multiple detection results, and reassemble the detection results to obtain the detection result of the fork groove edge in the box image.

[0070] Step S502: Generate a candidate edge line set according to the detection result.

[0071] The detection result of the fork groove edge in the box image output by the fork groove edge detection network is usually the pixel information of the fork groove edge. Figure 6An example of a detection result of the fork groove edge in the box image output by the fork groove edge detection network is shown. Among them, the first detection result example 610 and the second detection result example 620 both indicate the pixel information of the suspected fork groove edge on the box image, that is, each pixel of the suspected fork groove edge as the detection result has not yet formed a line segment. Therefore, after obtaining the detection result, the terminal 110 needs to further generate multiple edge lines (or edge segments) from the detection result. The edge lines combined according to the detection results are called candidate edge lines. Since there are multiple candidate edge lines, a candidate edge line set is formed. In a specific implementation, the terminal 110 may perform a Hough transform on the detection result to detect the straight lines in the detection result, so that the coordinate information of each straight line (or line segment) corresponding to the detection result can be obtained, thereby forming a candidate edge line set, such as Figure 7 The corresponding Figure 6 The straight line detection results are as follows, wherein the first candidate edge line set 710 corresponds to the candidate edge line set obtained after the first detection result example 610 is subjected to the Hough transform, and the second candidate edge line set 720 corresponds to the candidate edge line set obtained after the second detection result example 620 is subjected to the Hough transform.

[0072] Step S503, clustering the candidate edge line set according to the edge composition features of the box fork groove to obtain multiple categories of candidate edge lines;

[0073] Specifically, the candidate edge line set obtained in step S502 is actually fragmented, that is, the line segments have not been connected or merged. In this step, the terminal 110 needs to cluster the candidate edge line set obtained in step S502 according to the edge composition features of the box fork groove to obtain multiple categories of candidate edge lines. Among them, the edge composition features of the box fork groove refer to the features of the edge lines that constitute the fork groove, such as the composition features and geometric features of the edge lines of the fork groove. Exemplarily, the edge composition features are how many line segments the edge line of the fork groove is composed of, and what kind of geometric features each line segment has, such as whether it is horizontal or vertical, etc.

[0074] In some embodiments, step S503 specifically includes: clustering the candidate edge lines into first slope class candidate edge lines and second slope class candidate edge lines according to the edge composition features.

[0075] The edge line of the fork groove is generally composed of two vertical line segments and two horizontal line segments, so the terminal 110 can cluster the candidate edge line set obtained in step S502 according to the edge composition feature to obtain two types of candidate edge lines, one type is the vertical class candidate edge line, and the other type is the horizontal class candidate edge line. Specifically, the terminal 110 can obtain the slope of each candidate edge line in the candidate edge line set, and cluster these candidate edge lines into the first slope class candidate edge line and the second slope class candidate edge line according to the slope, wherein the first slope class candidate edge line can correspond to the vertical class candidate edge line, and the second slope class candidate edge line corresponds to the horizontal class candidate edge line. Among them, the candidate edge line with an absolute value of the slope greater than 3 is classified as the first slope class candidate edge line, and the candidate edge line with an absolute value of the slope less than 1 / 3 is classified as the second slope class candidate edge line.

[0076] Furthermore, after clustering the candidate edge lines into first slope class candidate edge lines and second slope class candidate edge lines, the following processing steps can be performed: based on the positional relationship between the candidate edge lines of the same type, the candidate edge lines included in the first slope class candidate edge lines are merged into first slope candidate edge lines, and the candidate edge lines included in the second slope class candidate edge lines are merged into second slope candidate edge lines; based on the positional relationship between the first slope candidate edge line and the second slope candidate edge line, the first slope candidate edge line and the second slope candidate edge line are spliced ​​to form a regional edge.

[0077] Among them, the same type of candidate edge lines refer to candidate edge lines that belong to the first slope class or the second slope class. Specifically, based on the positional relationship between the candidate edge lines of the first slope class, the terminal 110 can merge vertical lines with similar coordinates into one line. The similarity of coordinates can be determined by the distance between the endpoints of the line segments. If the distance between the endpoints of the line segments is less than a set threshold such as 20 pixels, the coordinates can be considered to be similar and merged into one line. All candidate edge lines of the first slope class are traversed to obtain the final merged result. Similarly, based on the positional relationship between the candidate edge lines of the second slope class, the terminal 110 can merge horizontal lines with similar coordinates into one line.

[0078] On this basis, we can further calculate the minimum horizontal coordinate xmin, the maximum horizontal coordinate xmax, the minimum vertical coordinate and the maximum vertical coordinate ymax corresponding to each type of candidate edge line as the coordinates of this type of straight line, that is, (xmin, ymin, xmax, ymax). If the length of the straight line is less than the set threshold such as 20 pixels, the straight line can be directly removed to improve the accuracy of determining the edge of the fork groove.

[0079] like Figure 8 Shown with Figure 7The merging result of the two types of candidate edge lines corresponding to the first candidate edge line set 710 and the second candidate edge line set 720, wherein, based on the positional relationship between the candidate edge lines of the first slope class, vertical straight lines with similar coordinates can be merged into first slope candidate edge lines 801, and based on the positional relationship between the candidate edge lines of the second slope class, horizontal straight lines with similar coordinates can be merged into second slope candidate edge lines 802. After obtaining the first slope candidate edge line 801 and the second slope candidate edge line 802, the terminal 110 can combine the horizontal straight lines and vertical straight lines with close coordinates according to the characteristics of the fork groove to obtain the regional edge of the fork groove candidate area. For example, the fork groove area is generally composed of two horizontal straight lines and two vertical straight lines. The terminal 110 can traverse all the first slope candidate edge lines 801 and the second slope candidate edge lines 802. If it is detected that the endpoint distance between a first slope candidate edge line 801 and a second slope candidate edge line 802 is less than a set threshold such as 20 pixels, the terminal 110 can combine the two candidate edge lines as components of the regional edge of the fork groove candidate area. If there are candidate edge lines that cannot constitute the regional edge, the terminal 110 can directly remove these candidate edge lines to improve the accuracy of determining the fork groove edge.

[0080] Step S504: forming a region edge based on the splicing of various candidate edge lines, and obtaining a plurality of fork groove candidate regions.

[0081] Among them, after the terminal 110 obtains the regional edge constituting the fork slot candidate area, a plurality of fork slot candidate areas can be formed. In some embodiments, the terminal 110 can also screen these fork slot candidate areas first. For the convenience of description, each suspicious fork slot area surrounded by the regional edge formed by the splicing of various candidate edge lines is called a fork slot preliminary selection area, and the terminal 110 can first perform a round of screening according to the regional size of these fork slot preliminary selection areas to obtain a plurality of fork slot candidate areas. Specifically, the terminal 110 first checks the regional size of each fork slot preliminary selection area, and for the fork slot preliminary selection area whose regional size is less than, for example, 300 pixels or greater than 700 pixels, the terminal 110 can exclude it, and the terminal 110 can further screen it according to the position of the fork slot preliminary selection area on the box image. If the fork slot preliminary selection area is too close to the left and right corner columns of the box or the center of the box, for example, the fork slot preliminary selection area and the left and right corner columns of the box or the center of the box are less than the set threshold, the terminal 110 can also exclude it, and finally screen out a plurality of fork slot candidate areas from the fork slot preliminary selection area to further improve the accuracy of fork slot area positioning.

[0082] The technical solution of the above embodiment can firstly vectorize the pixel points to obtain a candidate edge line set after detecting the fork groove edge through the fork groove edge detection network, and further, according to the edge composition characteristics of the box fork groove, each edge line in the candidate edge line set can be clustered according to the slope and then spliced ​​to accurately lock the edge of the area constituting the fork groove area.

[0083] In one embodiment, the terminal 110 may not input an entire box image into the fork groove edge detection network for fork groove edge detection, but may first extract a local box image containing the fork groove edge from the box image, and then perform fork groove edge detection on the local box image to improve the fork groove edge detection efficiency and positioning accuracy.

[0084] like Fig. 9 As shown, the detection result of the fork groove edge detection network on the fork groove edge in the box image in step S501 specifically includes:

[0085] Step S901, obtaining a local image of the box including the fork groove area from the box image according to the specification information of the box;

[0086] refer to Fig.10 , the terminal 110 obtains the box image that needs to be positioned in the fork groove area. The box image is a side image of the box, which mainly includes five parts: an upper beam, a lower beam, a side panel (the side panel is composed of a corrugated panel, and the corrugated panel includes a recessed area, a raised area, and a sloped area), a left corner column, and a right corner column. It can be noted that the fork groove is located in the lower beam. If the fork groove area is not positioned in the lower beam in the subsequent fork groove positioning, the box does not meet the specifications. In this step, the terminal 110 segments the box image according to the specification information of the box, and obtains a partial image of the box including the fork groove area from the box image. Fig.11 Shown with Fig.10 The corresponding example of segmentation of the box partial image is that the terminal 110 can segment the image of the lower beam portion from the box image as the box partial image including the fork groove area.

[0087] Step S902, dividing the box local image into a plurality of box local sub-images, and inputting the plurality of box local sub-images in parallel into a fork slot edge detection network, so that the fork slot edge detection network outputs a detection result corresponding to the fork slot edge in each box local sub-image;

[0088] Step S903, splicing the detection results corresponding to the edges of the fork grooves in the local sub-images of each box to obtain the detection results.

[0089] Before inputting the local image of the box into the fork groove edge detection network for fork groove edge detection, the terminal 110 may also evenly divide the local image of the box into a plurality of local sub-images of the box, and then input the plurality of local sub-images of the box into the fork groove edge detection network in parallel. The fork groove edge detection network performs fork groove edge detection on these local sub-images of the box in parallel, and outputs the detection results corresponding to the fork groove edges in each local sub-image of the box. Specifically, considering that the local image of the box after segmentation is generally still relatively large, generally with a width of 4000 to 6000 pixels, and the terminal 110 generally does not have so much video memory to process images of this size, in this step, a sliding window may be used to segment the local image of the box into a plurality of local sub-images of the box, such as 1024×512, and the plurality of local sub-images of the box are sent to the fork groove edge detection network in parallel. After the detection results of all the local sub-images of the box are obtained, the detection results of these local sub-images of the box are spliced ​​to obtain the segmentation results of the local image of the box.

[0090] By adopting the technical solution provided in this embodiment, the image area that needs to be detected for the fork groove area can be accurately located by segmenting the local image of the box, and the local image of the box can be further divided into small images to improve the detection efficiency of the fork groove edge detection network for the fork groove edge in the input image. Finally, accurate detection results can be obtained by splicing. More specifically, each detection result can be traversed through a sliding window, and a Hough transform can be performed on the detection result each time it is traversed. After completion, all straight line detection results of the local image of the box can be obtained.

[0091] For the fork slot edge detection network, it can be implemented based on the HRNet network. Among them, HRNet (High-Resoultion Net) is a high-resolution network, such as Fig.12 The original network structure diagram of the HRNet network is shown in Figure 2. For the application of the HRNet network in detection or segmentation, Fig.13 As shown, generally based on HRNet, the three feature maps (the first feature map 1301, the second feature map 1302 and the third feature map 1303) obtained by each resolution detection branch are upsampled to obtain three feature maps with consistent resolution, and then fused to obtain the final detection result 1304 output.

[0092] Considering the straight line characteristics of the fork groove edge, in some embodiments, such as Fig.14As shown, the fork groove edge detection network may include a first resolution detection branch, a second resolution detection branch and a third resolution detection branch. The fork groove edge detection network is used to fuse the feature maps output by the second resolution detection branch and the third resolution detection branch (including the second resolution detection branch feature map 1402 and the third resolution detection branch feature map 1403) to obtain a first fused feature map 1404, upsample the first fused feature map 1404 and fuse it with the feature map output by the first resolution detection branch (i.e., the first resolution detection branch feature map 1401) to obtain a second fused feature map 1405, and output the detection result according to the second fused feature map 1405. The technical solution provided in this embodiment does not require substantially any parameters for the network, and can achieve more accurate segmentation of the edge fork groove.

[0093] Combination Fig.15 The construction process of the fork slot edge detection network is explained as follows: Fig.15 , the above-mentioned box fork groove area determination method can also construct the fork groove edge detection network through the following steps:

[0094] Step S1501, obtaining a box sample image, and obtaining labeling information of the fork groove edge in the box sample image;

[0095] In this step, the terminal 110 may obtain box images serving as training samples of the fork groove edge detection network, referred to as box sample images, and then may further obtain labeling information for labeling the fork groove edges in these box sample images.

[0096] Step S1502, determining the area ratio of the fork groove area in the box sample image to the box sample image.

[0097] In this step, after obtaining the annotation information, the terminal 110 can determine the area ratio of the fork groove area in the box sample image to the box sample image according to the annotation information. In this embodiment, the area ratio is set to a, and the exemplary area ratio a can be 0.1.

[0098] Step S1503, determining the weight of the fork groove edge in the box sample image for the loss function of the fork groove edge detection network to be trained according to the area ratio, and constructing the loss function according to the weight

[0099] This step is mainly to determine the weight of the fork groove edge to the network's loss function based on the area ratio and construct the loss function based on this. Specifically, when annotating the fork groove area, each pixel in the image can be divided into two categories: background and edge straight line. Considering that the fork groove edge straight line occupies a smaller area of ​​the overall image, while the background is more, when calculating the loss function of the fork groove edge detection network, first calculate the area ratio of the fork groove area to the box sample image, and add the area ratio a (for example, a = 0.1) to the loss function. Then the weights of the background and straight line to the loss function of the fork groove edge detection network to be trained are a (i.e. 0.1) and 1-a (i.e. 0.9) respectively.

[0100] Step S1504, training the fork groove edge detection network to be trained using the box sample images, annotation information and loss function to construct the fork groove edge detection network.

[0101] After determining the specific form of the loss function, the terminal 110 or the server 120 can use the box sample images, the annotation information and the loss function to train the fork groove edge detection network to be trained so as to construct the fork groove edge detection network.

[0102] By adopting the technical solution provided in the above embodiment, the weight of the loss function of the fork groove edge detection network to be trained can be determined according to the area ratio of the fork groove area in the box sample image to the box sample image, and a loss function can be constructed based on this, so that the trained fork groove edge detection network can detect the fork groove edge with higher accuracy.

[0103] The number of fork groove candidate area groups obtained in step S203 is usually multiple, that is, multiple fork groove candidate area groups can usually be obtained before the fork groove area is finally confirmed. Specifically, the number of fork grooves on a general box body is usually two, so in some embodiments, the fork groove candidate area group includes a first fork groove candidate area and a second fork groove candidate area.

[0104] For the specific grouping process of the fork slot candidate regions, for example, the fork slot candidate regions to be grouped may be numbered as [X1, X2, ..., X n ], these regions are sorted in ascending order according to the horizontal coordinate values, and the region combinations (X1, X2), (X1, X3), … (X n ,X n-1 ), so as to make the final screening according to the size constraints and layout constraints formed by the standard size and layout of the fork slots of the box.

[0105] Furthermore, the standard size of the fork groove in the above step S204 may include a first standard size of the fork groove and a second standard size of the fork groove, that is, the size of the two fork grooves on the box under standard conditions. The standard layout of the fork groove includes a first reference distance between the first standard fork groove and the first box reference object, and a second reference distance between the second standard fork groove and the second box reference object.

[0106] Among them, for the convenience of description, it is assumed that the first standard fork groove is located on the left side of the box and the second standard fork groove is located on the right side of the box. The first box reference object can be the left corner column of the box, and the second box reference object can be the right corner column of the box. Then the first reference distance included in the fork groove standard layout is the distance between the fork groove on the left side of the box and the left corner column under standard conditions. Correspondingly, the second reference distance is the distance between the fork groove on the right side of the box and the right corner column under standard conditions.

[0107] Based on this, the above step S204 specifically includes:

[0108] If there is no image distortion in the box image, the size difference and distance difference corresponding to each fork groove candidate area group are obtained; and the fork groove area in the box image is determined from the fork groove candidate area group according to the size difference and distance difference;

[0109] The size difference includes the size difference between the area size of the first fork groove candidate area (referred to as the first area size) and the above-mentioned first fork groove standard size, and the size difference between the area size of the second fork groove candidate area (referred to as the second area size) and the above-mentioned second fork groove standard size. That is to say, in the case where there is no image distortion in the box image, the sizes of the two areas in each group should be close and close to the standard size specified by the fork groove. According to this constraint condition, the best area pair with the smallest difference from the standard size can be screened out. In this embodiment, the size difference is used as one of the screening condition constraints to accurately determine the fork groove area that meets the size requirements from the fork groove candidate area group.

[0110] In addition, the distance difference includes the distance difference between the first distance and the first reference distance, and the distance difference between the second distance and the second reference distance. The first distance refers to the distance between the first fork slot candidate area and the first box reference object (i.e., the left corner column), and the second distance refers to the distance between the second fork slot candidate area and the second box reference object (i.e., the right corner column). According to this constraint, the best area pair with the smallest distance difference can be screened out. In this embodiment, the distance difference is also used as one of the screening condition constraints to improve the accuracy of determining the fork slot area that meets the size requirements from the fork slot candidate area group.

[0111] If the box image has image distortion, the first ratio difference and the second ratio difference corresponding to each fork groove candidate area group are obtained; and the fork groove area in the box image is determined from the fork groove candidate area group according to the first ratio difference and the second ratio difference.

[0112] Among them, the first proportion difference refers to the difference between the first proportion and the first reference proportion, the first proportion refers to the ratio between the above-mentioned first distance and the first area size, and the first reference proportion is the ratio between the first reference distance and the first fork groove standard size; correspondingly, the second proportion difference refers to the difference between the second proportion and the second reference proportion, the second proportion refers to the ratio between the above-mentioned second distance and the second area size, and the second reference proportion is the ratio between the second reference distance and the second fork groove standard size.

[0113] In the case of image distortion in the box image, the first ratio difference and the second ratio difference corresponding to all fork slot candidate area groups are obtained. For example, for the fork slot candidate area group (X1, X2), the first distance d between the first fork slot candidate area X1 and the first box reference object (i.e., the left corner column) is first calculated. LX1 And the first area size of the first fork groove candidate area X1, such as the area width d X1 , set it as (d LX1 ,d X1 ), accordingly, the second fork groove candidate region X2 can be obtained (d LX2 ,d X2 ), and at the same time obtain the distance d from the first standard fork groove to the left corner column under standard conditions L , the distance d from the second standard fork groove to the right corner column R , and the area width d of the first and second standard fork grooves w On this basis, the first proportional difference can be calculated and the second proportional difference Traverse all fork groove candidate area groups, and the proportion difference of all fork groove candidate areas can be calculated. The two fork groove candidate areas with the smallest proportion difference are taken as the final two fork groove candidate areas. The proportion difference threshold can be further set to 1. If the proportion difference of the two fork groove candidate areas is the smallest and the proportion difference is within the threshold range, it means that the fork groove meets the standard, otherwise the fork groove does not meet the standard.

[0114] In some embodiments, it is also possible to determine whether the box image has image distortion in the following manner, specifically comprising: obtaining the relative size of the first distance and the second distance corresponding to each fork groove candidate area group, and determining whether the box image has image distortion.

[0115] This embodiment mainly determines whether there is image distortion in the box image based on the relative size of the above-mentioned first distance and the second distance corresponding to each fork slot candidate area group. Specifically, in the box standard, there are generally two fork slots, which are located on the lower beam of the box and are symmetrically distributed. The distance from the fork slot on the left side of the box to the left corner column and the distance from the fork slot on the left side of the box to the right corner column must be the same when there is no image distortion in the image. According to this principle, all fork slot candidate area groups can be traversed first to obtain the first distance and the second distance corresponding to each fork slot candidate area group. If there is a fork slot candidate area group with the same first distance and second distance among the fork slot candidate area groups, it is determined that there is no image distortion in the box image. If there is no fork slot candidate area group with the same first distance and second distance among the fork slot candidate area groups, it means that there is image distortion in the box image. In specific scenarios, there are many reasons for image distortion. One of the reasons is that when a container is transported into the yard, the image formed by the line scan camera will be distorted due to uneven road surface, turning, etc. The technical solution provided by the above embodiment can easily and accurately determine whether the image of the container is distorted, and can ensure that the position of the container fork slot can be accurately located even when the image is distorted. Finally, in the actual scenario, for further confirmation, the position of the fork slot area finally located on the container image can be displayed for manual review to ensure that there is no problem with the located fork slot.

[0116] like Figure 16(a) to Figure 16(c) The schematic diagram of the result of the present application for positioning the fork groove area in the box image is shown, wherein the 1610 part in FIG. 16 (a), the 1630 part in FIG. 16 (b) and the 1650 part in FIG. 16 (c) in these schematic diagrams are all line scan 2D images of the box, and the 1620 part in FIG. 16 (a), the 1640 part in FIG. 16 (b) and the 1660 part in FIG. 16 (c) are the corresponding fork groove area positioning results. It can be seen that the solution provided by the present application can obtain a relatively ideal positioning result regardless of the quality of the original image, especially for the box images shown in parts 1630 and 1650, which have relatively serious image distortion themselves, and the fork groove area in the box image shown in part 1650 is difficult to accurately locate even with the naked eye. Under this circumstance, the solution provided by the present application can still have an accurate positioning effect on the fork groove area.

[0117] Further, in conjunction with Figures 17(a) and 17(b), the effects of the method provided by the present application applied to the business process of container positioning are compared and explained. Among them, Figure 17(a) is the original business process for positioning the fork slot area of ​​the container, and Figure 17(b) is the business process for positioning the fork slot area of ​​the container using the method provided by the present application. By comparison, it can be seen that, unlike the original business process, the present application combines computer vision technology in the positioning of the fork slot area, and can also be assisted by less manual review on this basis. Moreover, the effect diagram of the corresponding positioning result also shows that the business process for positioning the fork slot area of ​​the container using the method provided by the present application has a faster and better positioning effect than the original business process.

[0118] The method provided in this application can realize the simultaneous completion of imaging and fork slot positioning of containers without the need for a vehicle while the containers are transported into the yard, which greatly reduces the time cost. In addition, only a line scan 2D camera is required for image acquisition of containers. The instrument device is relatively simple, which also reduces the equipment cost. This method can not only be applied to the fork slot positioning of containers in the yard, but also to the overall condition of the fork slots of containers that have just been produced and shipped to ensure the quality of the fork slots of containers. In addition, in the task of container deformation detection, the application of fork slot positioning can greatly improve the accuracy of deformation detection. This method also has the advantages of high robustness, fast positioning speed and high precision, which can greatly reduce the manual workload.

[0119] It should be understood that although Figure 2 to Figure 17(b) The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 to Figure 17(b) At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0120] In one embodiment, Fig.18 As shown, a device for determining a box fork slot area is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device 1800 specifically includes:

[0121] Image acquisition module 1801, used to acquire the box image;

[0122] The candidate determination module 1802 is used to determine a plurality of fork groove candidate regions in the box image by using a pre-built fork groove edge detection network; the region edges of the fork groove candidate regions are determined according to the detection result of the fork groove edge in the box image by the fork groove edge detection network;

[0123] A region combining module 1803 is used to combine the multiple fork groove candidate regions to obtain a fork groove candidate region group;

[0124] The area determination module 1804 is used to determine the fork groove area in the box image from the fork groove candidate area group based on the size constraints and layout constraints formed on the fork groove candidate area group by the fork groove standard size and fork groove standard layout of the box.

[0125] In one embodiment, the candidate determination module 1802 is further used to obtain the detection result of the fork groove edge in the box image by the fork groove edge detection network; generate a candidate edge line set based on the detection result; cluster the candidate edge line set according to the edge composition features of the box fork groove to obtain multiple categories of candidate edge lines; and obtain the multiple fork groove candidate areas based on the area edges formed by splicing various types of candidate edge lines.

[0126] In one embodiment, the candidate determination module 1802 is further used to cluster the candidate edge lines into first slope class candidate edge lines and second slope class candidate edge lines according to the edge composition features; the candidate determination module 1802 is further used to merge the candidate edge lines included in the first slope class candidate edge lines into first slope candidate edge lines and merge the candidate edge lines included in the second slope class candidate edge lines into second slope candidate edge lines based on the positional relationship between the same type of candidate edge lines before obtaining multiple fork groove candidate areas based on the area edges formed by splicing the various types of candidate edge lines; based on the positional relationship between the first slope candidate edge line and the second slope candidate edge line, splice the first slope candidate edge line and the second slope candidate edge line to form the area edge.

[0127] In one embodiment, the candidate determination module 1802 is further used to obtain a box local image including the fork groove area from the box image according to the specification information of the box; divide the box local image into multiple box local sub-images, and input the multiple box local sub-images in parallel to the fork groove edge detection network, so that the fork groove edge detection network outputs a detection result corresponding to the fork groove edge in each box local sub-image; and splice the detection results corresponding to the fork groove edge in each box local sub-image to obtain the detection result.

[0128] In one embodiment, the fork groove edge detection network includes a first resolution detection branch, a second resolution detection branch and a third resolution detection branch; the fork groove edge detection network is used to fuse the feature maps output by the second resolution detection branch and the third resolution detection branch to obtain a first fused feature map, upsample the first fused feature map and fuse it with the feature map output by the first resolution detection branch to obtain a second fused feature map, and output the detection result according to the second fused feature map.

[0129] In one embodiment, the above-mentioned device 1800 may also include: a network construction module, which is used to obtain a box sample image and obtain the annotation information of the fork groove edge in the box sample image; determine the area ratio of the fork groove area in the box sample image to the box sample image; determine the weight of the fork groove edge in the box sample image for the loss function of the fork groove edge detection network to be trained according to the area ratio, and construct the loss function according to the weight; use the box sample image, the annotation information and the loss function to train the fork groove edge detection network to be trained, and construct the fork groove edge detection network.

[0130] In one embodiment, the number of the fork groove candidate area groups is multiple; the fork groove candidate area groups include a first fork groove candidate area and a second fork groove candidate area; the fork groove standard size includes a first fork groove standard size and a second fork groove standard size; the fork groove standard layout includes a first reference distance between the first standard fork groove and a first box reference object and a second reference distance between the second standard fork groove and a second box reference object;

[0131] In one embodiment, the area determination module 1804 is further used to obtain the size difference and distance difference corresponding to each fork groove candidate area group if there is no image distortion in the box image; wherein the size difference includes the size difference between the first area size of the first fork groove candidate area and the first fork groove standard size and the size difference between the second area size of the second fork groove candidate area and the second fork groove standard size; the distance difference includes the distance difference between the first distance and the first reference distance and the distance difference between the second distance and the second reference distance; the first distance is the distance between the first fork groove candidate area and the first box reference object, and the second distance is the distance between the second fork groove candidate area and the second box reference object; determine the fork groove area in the box image from the fork groove candidate area group according to the size difference and the distance difference;

[0132] If there is image distortion in the box image, the first ratio difference and the second ratio difference corresponding to each fork groove candidate area group are obtained; wherein the first ratio difference is the difference between the first ratio and the first reference ratio, and the second ratio difference is the difference between the second ratio and the second reference ratio; the first ratio is the ratio of the first distance to the first area size, and the first reference ratio is the ratio of the first reference distance to the first fork groove standard size; the second ratio is the ratio of the second distance to the second area size, and the second reference ratio is the ratio of the second reference distance to the second fork groove standard size; the fork groove area in the box image is determined from the fork groove candidate area group according to the first ratio difference and the second ratio difference.

[0133] In one embodiment, the region determination module 1804 is further configured to obtain the relative size of the first distance and the second distance corresponding to each fork groove candidate region group, and determine whether the box image has image distortion.

[0134] The specific definition of the box fork groove area determination device can be found in the definition of the box fork groove area determination method above, which will not be repeated here. Each module in the above-mentioned box fork groove area determination device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0135] In one embodiment, a computer device is provided. The computer device may be a terminal or a server. The internal structure diagram thereof may be as follows: Fig.19 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with external devices through a network connection. When the computer program is executed by the processor, a method for determining a box fork slot area is implemented.

[0136] Those skilled in the art will understand that Fig.19 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0138] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0139] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.

[0140] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0141] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for determining a box fork slot area, characterized in that: The method comprises: Get the box image; Generate a candidate edge line set according to a detection result of the fork slot edge in the box image by a pre-built fork slot edge detection network; According to the edge composition features of the box fork groove, the candidate edge line set is clustered to obtain multiple categories of candidate edge lines; Based on the regional edges formed by splicing various candidate edge lines, multiple fork groove candidate regions are obtained; Combining the plurality of fork groove candidate regions to obtain a fork groove candidate region group; Based on the size constraints and layout constraints formed on the fork slot candidate area group by the fork slot standard size and the fork slot standard layout of the box, the fork slot area in the box image is determined from the fork slot candidate area group.

2. The method according to claim 1, characterized in that The method of clustering the candidate edge line set according to the edge formation features of the box fork groove to obtain multiple categories of candidate edge lines includes: According to the edge composition features, the candidate edge lines are clustered to form first slope class candidate edge lines and second slope class candidate edge lines; Before obtaining a plurality of candidate fork groove regions based on the regional edges formed by splicing various candidate edge lines, the method further includes: Based on the positional relationship between the candidate edge lines of the same type, the candidate edge lines included in the first slope type candidate edge lines are merged into first slope candidate edge lines, and the candidate edge lines included in the second slope type candidate edge lines are merged into second slope candidate edge lines; Based on the positional relationship between the first slope candidate edge line and the second slope candidate edge line, the first slope candidate edge line and the second slope candidate edge line are spliced ​​to form the region edge.

3. The method according to claim 1, characterized in that The method further comprises: According to the specification information of the box, a local image of the box including the fork groove area is acquired from the box image; Dividing the box local image into a plurality of box local sub-images, and inputting the plurality of box local sub-images into the fork slot edge detection network in parallel, so that the fork slot edge detection network outputs a detection result corresponding to the fork slot edge in each box local sub-image; The detection results corresponding to the edges of the fork grooves in the local sub-images of each box are spliced ​​to obtain the detection results.

4. The method according to any one of claims 1 to 3, characterized in that: The fork groove edge detection network includes a first resolution detection branch, a second resolution detection branch and a third resolution detection branch; the fork groove edge detection network is used to fuse the feature maps output by the second resolution detection branch and the third resolution detection branch to obtain a first fused feature map, upsample the first fused feature map and fuse it with the feature map output by the first resolution detection branch to obtain a second fused feature map, and output the detection result according to the second fused feature map.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Acquire a box sample image, and acquire labeling information of the fork groove edge in the box sample image; Determine the area ratio of the fork groove area in the box sample image to the box sample image; Determine the weight of the loss function of the fork groove edge in the box sample image for the fork groove edge detection network to be trained according to the area ratio, and construct the loss function according to the weight; The fork groove edge detection network to be trained is trained using the box sample image, annotation information and loss function to construct the fork groove edge detection network.

6. The method according to claim 1, characterized in that The number of the fork groove candidate area groups is multiple; the fork groove candidate area groups include a first fork groove candidate area and a second fork groove candidate area; the fork groove standard size includes a first fork groove standard size and a second fork groove standard size; the fork groove standard layout includes a first reference distance between the first standard fork groove and a first box reference object and a second reference distance between the second standard fork groove and a second box reference object; The step of determining the fork slot region in the box image from the fork slot candidate region group based on the size constraint and layout constraint formed on the fork slot candidate region group based on the fork slot standard size and fork slot standard layout of the box includes: If there is no image distortion in the box image, the size difference and distance difference corresponding to each fork groove candidate area group are obtained; wherein the size difference includes the size difference between the first area size of the first fork groove candidate area and the first fork groove standard size and the size difference between the second area size of the second fork groove candidate area and the second fork groove standard size; the distance difference includes the distance difference between the first distance and the first reference distance and the distance difference between the second distance and the second reference distance; the first distance is the distance between the first fork groove candidate area and the first box reference object, and the second distance is the distance between the second fork groove candidate area and the second box reference object; Determine the fork groove region in the box image from the fork groove candidate region group according to the size difference and the distance difference; If the box image has image distortion, obtain the first ratio difference and the second ratio difference corresponding to each fork groove candidate area group; wherein the first ratio difference is the difference between the first ratio and the first reference ratio, and the second ratio difference is the difference between the second ratio and the second reference ratio; the first ratio is the ratio of the first distance to the first area size, and the first reference ratio is the ratio of the first reference distance to the first fork groove standard size; the second ratio is the ratio of the second distance to the second area size, and the second reference ratio is the ratio of the second reference distance to the second fork groove standard size; A fork groove region in the box image is determined from the fork groove candidate region group according to the first ratio difference and the second ratio difference.

7. A device for determining a box fork groove area, characterized in that: The device comprises: An image acquisition module, used to acquire a box image; A candidate determination module is used to generate a candidate edge line set according to the detection result of the fork groove edge in the box image by the pre-built fork groove edge detection network; cluster the candidate edge line set according to the edge composition features of the box fork groove to obtain multiple categories of candidate edge lines; and obtain multiple fork groove candidate regions based on the regional edges formed by splicing various types of candidate edge lines; A region combination module, used for combining the plurality of fork groove candidate regions to obtain a fork groove candidate region group; The area determination module is used to determine the fork groove area in the box image from the fork groove candidate area group based on the size constraints and layout constraints formed on the fork groove candidate area group by the fork groove standard size and fork groove standard layout of the box.

8. The device according to claim 7, characterized in that The candidate determination module is further used to cluster the candidate edge lines into first slope class candidate edge lines and second slope class candidate edge lines according to the edge composition features; The candidate determination module is also used to merge the candidate edge lines included in the first slope class candidate edge lines into first slope candidate edge lines and merge the candidate edge lines included in the second slope class candidate edge lines into second slope candidate edge lines based on the positional relationship between the candidate edge lines of the same type before obtaining multiple fork groove candidate areas based on the area edge formed by splicing the candidate edge lines of various types; and splice the first slope candidate edge line and the second slope candidate edge line to form the area edge based on the positional relationship between the first slope candidate edge line and the second slope candidate edge line.

9. The device according to claim 7, characterized in that The candidate determination module is also used to obtain a box local image including the fork groove area from the box image according to the specification information of the box; divide the box local image into multiple box local sub-images, and input the multiple box local sub-images in parallel to the fork groove edge detection network, so that the fork groove edge detection network outputs a detection result corresponding to the fork groove edge in each box local sub-image; and splice the detection results corresponding to the fork groove edge in each box local sub-image to obtain the detection result.

10. The device according to any one of claims 7 to 9, characterized in that The fork groove edge detection network includes a first resolution detection branch, a second resolution detection branch and a third resolution detection branch; the fork groove edge detection network is used to fuse the feature maps output by the second resolution detection branch and the third resolution detection branch to obtain a first fused feature map, upsample the first fused feature map and fuse it with the feature map output by the first resolution detection branch to obtain a second fused feature map, and output the detection result according to the second fused feature map.

11. The device according to any one of claims 7 to 9, characterized in that The device also includes: A network construction module is used to obtain a box sample image and obtain the annotation information of the fork groove edge in the box sample image; determine the area ratio of the fork groove area in the box sample image to the box sample image; determine the weight of the fork groove edge in the box sample image for the loss function of the fork groove edge detection network to be trained according to the area ratio, and construct the loss function according to the weight; use the box sample image, the annotation information and the loss function to train the fork groove edge detection network to be trained, and construct the fork groove edge detection network.

12. The device according to claim 7, characterized in that The number of the fork groove candidate area groups is multiple; the fork groove candidate area groups include a first fork groove candidate area and a second fork groove candidate area; the fork groove standard size includes a first fork groove standard size and a second fork groove standard size; the fork groove standard layout includes a first reference distance between the first standard fork groove and a first box reference object and a second reference distance between the second standard fork groove and a second box reference object; The area determination module is further used for obtaining the size difference and distance difference corresponding to each fork groove candidate area group if there is no image distortion in the box image; wherein the size difference includes the size difference between the first area size of the first fork groove candidate area and the first fork groove standard size and the size difference between the second area size of the second fork groove candidate area and the second fork groove standard size; the distance difference includes the distance difference between the first distance and the first reference distance and the distance difference between the second distance and the second reference distance; the first distance is the distance between the first fork groove candidate area and the first box reference object, and the second distance is the distance between the second fork groove candidate area and the second box reference object; the fork groove area in the box image is determined from the fork groove candidate area group according to the size difference and the distance difference; The area determination module is also used to obtain the first ratio difference and the second ratio difference corresponding to each fork groove candidate area group if there is image distortion in the box image; wherein the first ratio difference is the difference between the first ratio and the first reference ratio, and the second ratio difference is the difference between the second ratio and the second reference ratio; the first ratio is the ratio of the first distance to the first area size, and the first reference ratio is the ratio of the first reference distance to the first fork groove standard size; the second ratio is the ratio of the second distance to the second area size, and the second reference ratio is the ratio of the second reference distance to the second fork groove standard size; the fork groove area in the box image is determined from the fork groove candidate area group according to the first ratio difference and the second ratio difference.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images