Method, apparatus, computer device, and storage medium for determining the air-permeable cover area of a box body

Abstract

This application relates to the field of computer vision technology, and provides a method, device, computer device, and storage medium for determining the air vent cover area of a box. This application can achieve efficient and accurate positioning of the air vent cover area in the box image, and also facilitate the comprehensive and effective monitoring of whether the air vent covers equipped on the box meet industry standards. The method includes: first, obtaining the box image and its corresponding depth image, then determining multiple air vent cover candidate areas in the box image according to the depth information provided by the depth image, then inputting the candidate area images corresponding to the multiple air vent cover candidate areas into the air vent cover area detection network, so that the air vent cover area detection network outputs the detection results corresponding to these candidate area images, and finally, the air vent cover area of the box can be determined from the above multiple air vent cover candidate areas according to the detection results.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and particularly to a method and device for determining the ventilation cover area of a box body, as well as a computer device and a storage medium. Background Art

[0002] With the rapid development of logistics technology, currently, boxes such as containers with certain strength, stiffness, and specifications are usually used for loading, storing, and transporting goods. The ventilation cover is a component of boxes such as containers. If the ventilation cover of the box does not meet the specifications or is even missing, it will lead to poor air permeability inside the box, and it is very likely to cause significant damage to the transported goods.

[0003] In the current technology, for the position of the ventilation cover of the box body, manual measurement is often required, which takes a lot of time and has low efficiency. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a method and device for determining the ventilation cover area of a box body, as well as a computer device and a storage medium.

[0005] A method for determining the ventilation cover area of a box body, the method comprising:

[0006] Obtaining a box body image and obtaining a depth image corresponding to the box body image;

[0007] Determining a plurality of ventilation cover candidate regions in the box body image according to the depth information provided by the depth image;

[0008] Inputting the candidate region image corresponding to each ventilation cover candidate region into a pre-constructed ventilation cover region detection network, so that the ventilation cover region detection network outputs a detection result corresponding to each candidate region image;

[0009] Determining the ventilation cover area of the box body from the plurality of ventilation cover candidate regions according to the detection result.

[0010] A device for determining the ventilation cover area of a box body, the device comprising:

[0011] An image acquisition module, configured to acquire a box body image and acquire a depth image corresponding to the box body image;

[0012] A candidate determination module, configured to determine a plurality of ventilation cover candidate regions in the box body image according to the depth information provided by the depth image;

[0013] A region detection module, configured to input the candidate region image corresponding to each ventilation cover candidate region into a pre-constructed ventilation cover region detection network, so that the ventilation cover region detection network outputs a detection result corresponding to each candidate region image;

[0014] An area determination module, configured to determine the ventilation cover area of the box body from the multiple ventilation cover candidate areas according to the detection result.

[0015] A computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0016] Obtain a box body image and a depth image corresponding to the box body image; determine multiple ventilation cover candidate areas in the box body image according to the depth information provided by the depth image; input the candidate area images corresponding to the ventilation cover candidate areas into a pre-constructed ventilation cover area detection network, so that the ventilation cover area detection network outputs the detection results corresponding to the candidate area images; determine the ventilation cover area of the box body from the multiple ventilation cover candidate areas according to the detection result.

[0017] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0018] Obtain a box body image and a depth image corresponding to the box body image; determine multiple ventilation cover candidate areas in the box body image according to the depth information provided by the depth image; input the candidate area images corresponding to the ventilation cover candidate areas into a pre-constructed ventilation cover area detection network, so that the ventilation cover area detection network outputs the detection results corresponding to the candidate area images; determine the ventilation cover area of the box body from the multiple ventilation cover candidate areas according to the detection result.

[0019] For the above-mentioned method, device, computer device and storage medium for determining the ventilation cover area of the box body, first obtain the box body image and its corresponding depth image, then determine multiple ventilation cover candidate areas in the box body image according to the depth information provided by the depth image, and then input the candidate area images corresponding to the multiple ventilation cover candidate areas into the ventilation cover area detection network, so that the ventilation cover area detection network outputs the detection results corresponding to these candidate area images, and finally the ventilation cover area of the box body can be determined from the multiple ventilation cover candidate areas according to the detection result. This solution can first use the depth information provided by the depth image to preliminarily screen the ventilation cover area as a candidate area, and then input the images corresponding to these candidate areas into the ventilation cover area detection network, and further accurately determine the true ventilation cover area on the box body through this network, so as to achieve efficient and accurate positioning of the ventilation cover area in the box body image. This solution can be implemented based on a computer device to facilitate comprehensive and effective monitoring of whether the ventilation cover equipped on the box body meets industry standards. Description of the Drawings

[0020] Figure 1Application environment diagram of the method for determining the air-permeable cover area of the box body in an embodiment;

[0021] Figure 2 Flow schematic diagram of the method for determining the air-permeable cover area of the box body in an embodiment;

[0022] Figure 3(a) is a schematic diagram of the box body image in an embodiment;

[0023] Figure 3(b) is a schematic diagram of the depth image in an embodiment;

[0024] Figure 3(c) is a schematic diagram of the positioning result of the air-permeable cover area in an embodiment;

[0025] Figure 4 Flow schematic diagram of the steps for determining the candidate area of the air-permeable cover in the box body image according to the depth information in an embodiment;

[0026] Figure 5(a) is a schematic diagram of the box body image structure in an embodiment;

[0027] Figure 5(b) is a schematic diagram of the depth image in an embodiment;

[0028] Figure 6(a) is a schematic diagram of the local image of the box body in an embodiment;

[0029] Figure 6(b) is a schematic diagram of the local depth image in an embodiment;

[0030] Figure 7(a) is a schematic diagram of the groove area in the local image of the box body in an embodiment;

[0031] Figure 7(b) is a schematic diagram of the groove area in the local depth image in an embodiment;

[0032] Figure 7(c) is a schematic diagram of the positioning result of the candidate area of the air-permeable cover in an embodiment;

[0033] Figure 7(d) is a schematic diagram of the positioning result of the air-permeable cover area in the local image of the box body in an embodiment;

[0034] Figure 8 Schematic diagram of the positioning result of the air-permeable cover area in the box body image in an embodiment;

[0035] Figure 9(a) is a schematic diagram of the original structure of the ResNet18 network;

[0036] Figure 9(b) is a schematic diagram of the structure of the air-permeable cover area detection network in an embodiment;

[0037] Figure 10(a) is a schematic diagram of the positioning result of the air-permeable cover area in the box body image in an embodiment;

[0038] Figure 10(b) is a schematic diagram of the positioning result of the ventilation cover area in the box body image in another embodiment;

[0039] Figure 10(c) is a schematic diagram of the positioning result of the ventilation cover area in the box body image in yet another embodiment;

[0040] Figure 11(a) is a schematic diagram of the business process for originally positioning the ventilation cover area of the container;

[0041] Figure 11(b) is a schematic diagram of the business process for positioning the ventilation cover area of the container in an embodiment;

[0042] Figure 12 is a structural block diagram of a device for determining the ventilation cover area of the box body in an embodiment;

[0043] Figure 13 is an internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0044] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] Some of the abbreviations and term definitions involved in the present application are as follows:

[0046] Linear array scanning imaging: Linear array scanning imaging means that when imaging an object, each time a line image is formed by line scanning on the image plane, or even a two-dimensional image, and finally, along the direction of the object's movement, stitching is performed to achieve complete imaging, which is abbreviated as line scanning imaging.

[0047] 2D image: A 2D image is also called a planar graph. A 2D image only has the X-axis and the Y-axis.

[0048] Depth image: The gray value of each pixel point in the depth image can be used to represent the distance of a certain point in the scene from the camera.

[0049] ResNet: A type of network based on deep learning.

[0050] Ventilation cover: A component on a box body such as a container, and can also be called a ventilation hood, ventilation hole, ventilator or ventilation opening, etc. The external dimensions of the ventilation cover are generally 200 mm × 75 mm × 25 mm, and its main function is to achieve the air permeability inside the box.

[0051] The automatic positioning method for the ventilation cover area of the box body provided by the present application can be combined with computer vision technology, and the 2D image of the box body and the corresponding depth image are used to efficiently and accurately position the ventilation cover area of the box body.

[0052] The method for determining the air vent cover area of the box provided in this application can be applied to, for example, Figure 1 the application environment shown. In this application environment, the terminal 110 communicates with the server 120 through a network. Among them, the terminal 110 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 110 and the server 120 can be directly or indirectly connected through wired or wireless communication methods, and this application does not limit this.

[0053] Specifically, the method for determining the air vent cover area of the box provided in this application can be executed by the terminal 110 or by the server 120, or can be executed in cooperation by the terminal 110 and the server 120. As an example, the following will be described separately by taking the execution by the terminal 110 and the execution in cooperation by the terminal 110 and the server 120 as examples.

[0054] In an exemplary embodiment, the terminal 110 acquires the box image and its corresponding depth image, then determines multiple air vent cover candidate areas in the box image according to the depth information provided by the depth image. Then, the terminal 110 inputs the candidate area images corresponding to each air vent cover candidate area into a pre-constructed air vent cover area detection network, so that the air vent cover area detection network outputs the detection results corresponding to each candidate area image. Finally, the air vent cover area of the box is determined from the above-mentioned multiple air vent cover candidate areas according to these detection results. This exemplary technical solution can be independently executed by the terminal 110 at the site where the box is located without the need to rely on the server 120, that is, the terminal 110 can complete the accurate and efficient positioning of the air vent cover area of the box without connecting to the Internet, is not easily affected by the stability of the Internet, and has high reliability.

[0055] In another exemplary embodiment, the terminal 110 may acquire a box body image and its corresponding depth image. Then, the terminal 110 may send the box body image and the depth image to the server 120. After receiving the box body image and the depth image, the server 120 may determine multiple ventilation hood candidate regions in the box body image according to the depth information provided by the depth image. Next, the server 120 inputs the candidate region images corresponding to the respective ventilation hood candidate regions into a pre-constructed ventilation hood region detection network, so that the ventilation hood region detection network outputs detection results corresponding to the respective candidate region images. The server 120 can then determine the ventilation hood region of the box body from the above-mentioned multiple ventilation hood candidate regions according to these detection results, and feedback the positioning result of the ventilation hood region of the box body to the terminal 110. This exemplary technical solution can be executed in cooperation by the terminal 110 located at the site where the box body is located and the remote server 120. This solution can reduce the data processing pressure on the terminal 110. Moreover, this solution provides the positioning service of the ventilation hood region by the server 120, that is, the server 120 can be used as the providing end of the artificial intelligence cloud service AIaaS (AI as a Service, Chinese for "AI as a Service"). Among them, the artificial intelligence cloud service can split various AI services and provide independent or packaged services in the cloud. Specifically in this embodiment, the server 120 provides an AI service for the terminal 110 to locate the ventilation hood region based on the ventilation hood region detection network. The terminal 110 can access and use this kind of artificial intelligence service provided by the server 120 through the API interface. The terminal 110 can also use the server 120 to deploy and operate and maintain a dedicated cloud artificial intelligence service, so as to comprehensively and effectively monitor whether the ventilation hood of the box body meets industry specifications.

[0056] In one embodiment, as Figure 2 shown, a method for determining a ventilation hood region of a box body is provided. Taking the method applied to the Figure 1 terminal 110 in it as an example for description, it includes the following steps:

[0057] Step S201, acquire a box body image and acquire a depth image corresponding to the box body image;

[0058] Among them, the box image refers to an image of a box that requires positioning of the breather hood area. This image can be a 2D image of the box to be photographed captured by a line-scan camera. The gray value of each pixel point in the depth image corresponding to the box image can be used to represent the distance of each point on the box to be photographed from the camera. The depth image corresponding to the box image can provide the depth information corresponding to each point on the box. Exemplarily, as shown in FIG. 3(a), it is a line-scan 2D gray image of a container side plate, and FIG. 3(b) shows the corresponding depth image. For the convenience of intuitively understanding the effect achieved by the breather hood area determination method provided in this application, the positioning result 310 of the breather hood area on the line-scan 2D gray image of the container side plate corresponding to FIG. 3(a) is shown in FIG. 3(c).

[0059] Step S202: Determine multiple breather hood candidate areas in the box image according to the depth information provided by the depth image;

[0060] In this step, the terminal 110 first preliminarily screens out multiple areas where the breather hood may be located from the box image as the breather hood candidate areas according to the depth information corresponding to each point on the box provided by the depth image.

[0061] Step S203: Input the candidate area images corresponding to the breather hood candidate areas into a pre-constructed breather hood area detection network, so that the breather hood area detection network outputs the detection results corresponding to the candidate area images;

[0062] In this step, after the terminal 110 completes the preliminary screening of the multiple breather hood candidate areas, it can further use the pre-constructed breather hood area detection network for further screening. Among them, the breather hood area detection network refers to a neural network that has the ability to identify or detect whether an image is a breather hood area. Specifically, the terminal 110 can obtain the candidate area images corresponding to the breather hood candidate areas. Among them, the candidate area image can include the image corresponding to the multiple candidate area images in the box image (referred to as the first image), or can include the image corresponding to the multiple candidate area images in the depth image (referred to as the second image). The first image or the second image or the combination of the first image and the second image can be used as the input image of the pre-constructed breather hood area detection network. That is, the terminal 110 can separately use or simultaneously use the images corresponding to the multiple candidate area images in the box image and the depth image as the input data for the breather hood area detection network to detect the breather hood area. Since the box image and the depth image correspond to each other, after the breather hood area detection network outputs the detection result of the breather hood area of the box image or the depth image, the detection result of the breather hood area on the other image can be obtained.

[0063] If both the first image and the second image corresponding to the box image and the depth image are used as input data, the accuracy of detecting the ventilation hood area of each candidate area image by the ventilation hood area detection network can be improved. Exemplarily, the terminal 110 can convert both the first image and the second image corresponding to the box image and the depth image into single-channel images, and then combine them into a three-channel image as the input image of the ventilation hood area detection network. Among them, the three-channel image can be the 2D grayscale image of the first image, the 2D grayscale image of the first image, and the second image. Then, the terminal 110 can input the three-channel image into the pre-constructed ventilation hood area detection network, and the ventilation hood area detection network outputs the detection result of the corresponding ventilation hood area according to the input image, that is, determines whether the input image corresponds to the real ventilation hood area.

[0064] Step S204, according to the detection result, determine the ventilation hood area of the box from multiple ventilation hood candidate areas.

[0065] The terminal 110 can obtain the detection result of each candidate area image output by the ventilation hood area detection network. If the detection result output by the ventilation hood area detection network shows that the input candidate area image is the ventilation hood area, the area on the box image corresponding to the candidate area image is determined as the ventilation hood area, so as to realize the recognition and positioning of the ventilation hood area. An example of the final positioning result of the ventilation hood area can be referred to Fig. 3(c).

[0066] In the above method for determining the ventilation hood area of the box, the terminal 110 first obtains the box image and its corresponding depth image, and then the terminal 110 determines multiple ventilation hood candidate areas in the box image according to the depth information provided by the depth image. Then, the terminal 110 inputs the candidate area images corresponding to the multiple ventilation hood candidate areas into the ventilation hood area detection network, so that the ventilation hood area detection network outputs the detection results corresponding to these candidate area images. Finally, the terminal 110 can determine the ventilation hood area of the box from the above multiple ventilation hood candidate areas according to the detection result. This solution can first use the depth information provided by the depth image to preliminarily screen the ventilation hood area as a candidate area, and then input the images corresponding to these candidate areas into the ventilation hood area detection network to further accurately determine the real ventilation hood area on the box through this network, so as to realize the efficient and accurate positioning of the ventilation hood area in the box image. This solution can be implemented based on a computer device to facilitate the comprehensive and effective monitoring of whether the ventilation hood equipped on the box meets the industry standards.

[0067] The following combines Figure 4 to illustrate the step of determining multiple ventilation hood candidate areas in the box image according to the depth information provided by the depth image in the foregoing step S202, as Figure 4 shown, this step S202 may specifically include:

[0068] Step S401: Determine multiple groove regions in the box image according to the depth information.

[0069] Specifically, the ventilation hood of the box is located in the groove region of the box. Taking a container as an example, the ventilation hood on the container is located in the recessed region of the container corrugated board, and the depth value of the groove region is generally the largest on the depth image. In this way, all groove regions can be obtained. In this step, the terminal 110 can first determine multiple groove regions in the box image according to the depth information provided in the depth image.

[0070] Step S402: Determine the preliminary selection regions of the ventilation hoods in each groove region based on the depth value distribution of the groove regions.

[0071] Considering that the ventilation hood is a convex shape, the depth value corresponding to the region where the ventilation hood is located is different from other positions in the groove region. Compared with the region without a ventilation hood, its depth value generally becomes smaller. Based on this, in this step, the depth value distribution of the groove region can be obtained first. This depth value distribution can include the depth values of each position or region in each groove region, so that the terminal 110 can initially locate the region in each groove region where the depth value is different from the depth values of other positions as the preliminary selection region of the ventilation hood.

[0072] In some embodiments, step S402 may specifically include:

[0073] Based on the depth value distribution of the groove region, determine whether there is a depth anomaly region in the groove region; if there is a depth anomaly region in the groove region, then use the depth anomaly region as the preliminary selection region of the ventilation hood.

[0074] Among them, the depth anomaly region is a region that forms a preset depth difference with the adjacent region, and the adjacent region refers to the region adjacent to the depth anomaly region in the above groove region. Specifically, the ventilation hood is a cover, which generally protrudes outwards, so the depth value of the ventilation hood region is relatively smaller than the depth values of other positions in the groove region. Taking advantage of this, the terminal 110 can traverse the depth map of each groove region from top to bottom. When traversing, if it is detected that a certain region of the groove region is relatively smaller in depth value than its adjacent regions, it can be regarded that this region forms a preset depth difference with its adjacent regions. Thus, the terminal 110 can determine that this depth anomaly region is the preliminary selection region of the ventilation hood. Using the preliminary selection region positioning scheme provided in this embodiment can accurately and quickly initially lock the position where the ventilation hood region is located.

[0075] Step S403: Obtain multiple ventilation hood candidate regions from each initial candidate region of the ventilation hood according to the region size of each preliminary selection region of the ventilation hood.

[0076] In this step, after the terminal 110 obtains multiple primary selection areas of the ventilation hoods, it can further screen according to the area sizes of the primary selection areas of the ventilation hoods to obtain multiple candidate areas of the ventilation hoods.

[0077] In some embodiments, step S403 may specifically include: regarding the primary selection areas of the ventilation hoods with area sizes within a preset area size range as the candidate areas of the ventilation hoods.

[0078] In this embodiment, after the terminal 110 obtains multiple primary selection areas of the ventilation hoods, it can compare the area size of each primary selection area of the ventilation hood with the preset area size range. Exemplarily, if the area length of the primary selection area of the ventilation hood is greater than 50 pixels and less than 300 pixels, then this primary selection area of the ventilation hood can be regarded as a candidate area of the ventilation hood. On this basis, the images corresponding to the candidate areas of the ventilation hood can be further cut out from the box image and the depth image as candidate area images. The solution provided in this embodiment can, on the basis of using depth information for preliminary positioning, further screen the area where the ventilation hood is located through the detection of whether the area size is within the preset area size range, so as to improve the positioning accuracy of the ventilation hood area.

[0079] In some embodiments, before positioning the primary selection areas of the ventilation hoods, the box image and the depth image can also be divided first, so that the position where the ventilation hood is located can be locked more quickly and effectively in the detection process of the primary selection areas of the ventilation hoods, the candidate areas, and the ventilation hood area detection network for the ventilation hood area.

[0080] As Figure 5(a) and 5(b) shown are the box image and its corresponding depth image. The terminal 110 acquires the box image and its corresponding depth image that need to perform ventilation hood area positioning. The box image can be, for example, a 2D line scan image of the side of a box such as a container, and the depth image is the depth image of the same area corresponding to it. The box shown in the box image mainly includes five parts: the upper side beam, the lower side beam, the side plate (the side plate is composed of corrugated plates, and the corrugated plates include a concave area, a convex area, and an inclined surface area), the left corner post, and the right corner post, where the ventilation hood is located above the side plate. Based on this, for the box image and its corresponding depth image, the terminal 110 can cut the box image and its corresponding depth image according to the specification information of the box to obtain a local box image and a local depth image containing the ventilation hood area. As shown in FIGS. 6(a) and 6(b), the local image segmentation results corresponding to Figure 5(a) and 5(b) are shown.

[0081] On this basis, the terminal 110 can use the local box image and the local depth image to perform positioning of the primary selection areas, screening of the candidate areas, and detection processing of the ventilation hood area using the ventilation hood area detection network. As Figures 7(a) to 7(d), wherein, FIGS. 7(a) and 7(b) show multiple groove regions 710 in the local image of the box body determined according to the depth information and multiple groove regions 720 in the local depth image. The initial selection regions of the ventilation covers in each groove region can be further determined according to the depth value distribution of the groove regions, and multiple candidate regions of the ventilation covers can be selected therefrom according to the region sizes of the initial selection regions of the ventilation covers. As shown in FIG. 7(c), the first candidate region 731 of the ventilation cover, the second candidate region 732 of the ventilation cover, and the third candidate region 733 of the ventilation cover are shown. These candidate regions of the ventilation covers can all be used as suspected ventilation cover regions and can be cut out from the local image of the box body and the local depth image to obtain candidate region images, and then input into the ventilation cover region detection network, so that the ventilation cover region detection network outputs detection results corresponding to these candidate region images respectively, that is, outputs which of these candidate region images are ventilation covers and which are not. As shown in FIG. 7(d), the ventilation cover region detection network shows the detection results of whether the candidate region images corresponding to the first candidate region 731 of the ventilation cover, the second candidate region 732 of the ventilation cover, and the third candidate region 733 of the ventilation cover in FIG. 7(c) are ventilation covers. The detection results show that neither the first candidate region 731 of the ventilation cover nor the second candidate region 732 of the ventilation cover is a ventilation cover, and only the third candidate region 733 of the ventilation cover is the region where the ventilation cover is located.

[0082] The above process can locate the specific position of the ventilation cover region of the box body in the local image of the box body. Then, the terminal 110 can map the pixel coordinates corresponding to the local image of the box body back to the original box body image, and thus obtain the pixel coordinates of the ventilation cover region on the box body image. At the same time, in combination with the imaging parameters of the camera, the pixel length and width of the ventilation cover region in the box body image are converted into the actual length and width (unit: millimeter), and using the depth image, the height of the ventilation cover region can also be obtained. In this way, the specific size information such as the length, width, and height of the ventilation cover region can be obtained and compared with the standard size of the ventilation cover to confirm whether the ventilation cover meets the specifications.

[0083] On this basis, the terminal 110 can, according to the specific pixel coordinates of the ventilation cover region on the box body image, such as Figure 8 FIG. shows the positioning result 810 of the ventilation cover region on the box body image. In combination with the imaging parameters of the camera, the terminal 110 can further convert the pixel length and width of the ventilation cover region in the box body image into the actual length and width (unit: millimeter) of the ventilation cover region. Using the depth image, the height of the ventilation cover region can be correspondingly obtained, so that the specific size information such as the length, width, and height of the ventilation cover region can be obtained and compared with the standard size of the ventilation cover region to confirm whether the ventilation cover meets the specifications.

[0084] Specifically, after locating the specific position of the box ventilation cover area on the local image, since the coordinates of the segmented local image in the box image can be recorded during the segmentation of the local image, using this, the specific position of the box ventilation cover area located on the local image can be mapped back to the original box image, and the specific pixel coordinates of the ventilation cover area can be obtained on the box image. Next, in combination with the imaging parameters of the camera, the pixel length and width of the ventilation cover area in the box image can be converted into actual length and width. Using the depth image, the depth of the ventilation cover area and its corresponding groove area can be specifically obtained, and thus the height of the ventilation cover area can be obtained. In this way, the specific length, width, and height information of the ventilation cover area can be obtained and compared with the standard size of the ventilation cover area to confirm whether the ventilation cover meets the specifications. If the terminal 110 fails to locate the ventilation cover area on the box image, it can be determined that the box is missing a ventilation cover.

[0085] Considering that the captured image may be distorted, direct comparison with the standard may cause problems. In actual production, the distortion is basically caused by the turning of the container and the uneven road surface. During actual calculation, the length resolution of the ventilation cover (unit: mm / pixel) is obtained by converting the depth value and the camera parameters. The height value of the ventilation cover can be obtained by subtracting the depth value of the groove where the ventilation cover is located from the depth value of the ventilation cover area. Turning will change the overall depth value of the container, but for a single groove, the width is small, and the influence of turning can be ignored, and the depth value of the area remains basically unchanged. Moreover, the uneven road surface does not affect the resolution in the length and height directions. Then, the influence of image distortion on the height and length of the ventilation cover can be ignored. Under normal circumstances, the width of the ventilation cover is basically the same as the width of the groove. By comparing the pixel width of the ventilation cover with the pixel width of the groove where it is located, if the width difference is within 5 pixels, it can be considered that the width of the ventilation cover meets the specifications. In the actual scenario, for further confirmation, the identified ventilation cover area can be manually reviewed to ensure that there is no problem with the located ventilation cover.

[0086] For the ventilation cover area detection network, it can be implemented based on the ResNet18 network. As shown in Fig. 9(a), it is the structural schematic diagram of the original ResNet18 network. As shown in Fig. 9(b), considering the characteristics of the ventilation cover area itself, the original ResNet18 network is improved to a certain extent. First, a branch (referred to as the branch network) is led out from the 13th layer of the main path network of the ResNet18 network, keeping the resolution and the number of channels unchanged. The structure of the branch network is similar to that of the 10th to 13th layers of the main path network. For the feature map output by the fourth layer of the branch network, 3×3 convolution operations can be used to change the number of channels and the length and width of the output feature map, so that it is consistent with the size and number of channels of the feature map of the main path network. Finally, the ventilation cover area detection network composed of this main path and branch network structure can fuse the outputs of the two paths to output the detection result.

[0087] Based on this, in one embodiment, inputting the candidate region images corresponding to each breathable mask candidate region into a pre-constructed breathable mask region detection network, so that the breathable mask region detection network outputs detection results corresponding to each candidate region image, specifically includes:

[0088] Input the candidate region images corresponding to each breathable mask candidate region into the breathable mask region detection network, so that the breathable mask region detection network obtains a first feature map corresponding to the candidate region image through the main path network and a second feature map corresponding to the candidate region image through the branch network, fuse the first feature map and the second feature map to obtain a fused feature map, and output the detection result according to the fused feature map.

[0089] In this embodiment, the specific structure of the breathable mask region detection network refers to FIG. 9(b). The breathable mask region detection network includes a main path network and a branch network. After the candidate region image is input into the breathable mask region detection network, the main path network of the breathable mask region detection network outputs the corresponding first feature map, and the branch network outputs the corresponding second feature map. For the feature map output by the branch network, the number of channels and the length and width of the feature map can be made consistent with those of the first feature map of the main path network by using a 3×3 convolution operation. Finally, the first feature map and the second feature map are fused to obtain a fused feature map, and the corresponding detection result is output. The technical solution provided in this embodiment can, by introducing a detection branch, on the one hand, share some features extracted by the main path, and on the other hand, perform richer feature detection in the branch network, improving the accuracy of the breathable mask region detection network for detecting the breathable mask region.

[0090] Further, the fusion method of the breathable mask region detection network for the first and second feature maps can be based on the weights corresponding to the first and second feature maps. Specifically, in some embodiments, the breathable mask region detection network can be further configured to fuse the first feature map and the second feature map according to the first fusion weight corresponding to the first feature map and the second fusion weight corresponding to the second feature map to obtain the above-mentioned fused feature map. Among them, let the first feature map output by the main path network be w1, and the second feature map output by the branch network be w2. Then the fusion method can be that the fused feature map w = A1×w1 + A2×w2, and then a 1×1 convolution layer is used to replace the fully connected layer of the original ResNet18 network to perform operations on the fused feature map to output the detection result. Among them, A1 and A2 respectively represent the weights corresponding to the first and second feature maps, and a weight greater than that of the branch network can be assigned to the main path network. Exemplarily, the weight A1 corresponding to the first feature map can be taken as 0.7, and the weight A2 corresponding to the second feature map can be taken as 0.3.

[0091] The following describes the construction process of the breathable hood area detection network by way of examples. In some embodiments, the breathable hood area detection network can be constructed using the following steps, which specifically include:

[0092] Obtain the box sample image and the corresponding depth sample image of the box sample image; obtain the positive sample area image corresponding to the breathable hood area from the box sample image and the depth sample image, and obtain the negative sample area image from the groove area of the box sample image and the depth sample image; use a preset proportion of the positive sample area image and the negative sample area image to train the breathable hood area detection network to be trained, and construct the breathable hood area detection network.

[0093] Specifically, first obtain a batch of box 2D grayscale images with marked breathable hood coordinates and the corresponding depth images as the box sample image and the depth sample image. Then, according to the marked breathable hood coordinates, cut out the area image corresponding to the breathable hood area from the box sample image and the depth sample image, which is called the positive sample area image. At the same time, cut out some small images at other positions in the groove area as the negative sample area image, and this other position refers to the position in the groove area that is different from the breathable hood area. Next, these positive sample area images and negative sample area images can be preprocessed, which can include converting the corresponding 2D grayscale image and depth image into a single-channel image first, and then splicing them into a three-channel image. Each channel of the three-channel image is the 2D grayscale image, the 2D grayscale image, and the depth image respectively, and the length and width can be uniformly converted to 448 pixels and 224 pixels. These preprocessed positive and negative sample area images will be used as training data. On this basis, the positive and negative sample ratios can also be set. For example, the ratio of the number of positive samples and negative samples output to the breathable hood area detection network can be kept at 1:1. After starting the training, continuously iterate and optimize the detection results until the breathable hood area detection network meets the usage requirements. In this way, a breathable hood area detection network model that can accurately detect the breathable hood area can be obtained. When the network model is actually applied, the initially screened candidate area image can be preprocessed in the same way and then sent into the breathable hood area detection network to obtain whether the area is a real breathable hood area.

[0094] Such as Figures 10(a) to 10(c)The figure shows a schematic diagram of the result of positioning the ventilation hood area in the box image in the present application. Among them, the 1010 part in Fig. 10(a), the 1040 part in Fig. 10(b), and the 1070 part in Fig. 10(c) in these schematic diagrams all show the line-scanned 2D images of the box, the 1020 part in Fig. 10(a), the 1050 part in Fig. 10(b), and the 1080 part in Fig. 10(c) show the depth images of the corresponding boxes, and the 1030 part in Fig. 10(a), the 1060 part in Fig. 10(b), and the 1090 part in Fig. 10(c) show the positioning results of the ventilation hood areas of the corresponding boxes. It can be seen that for the solution provided by the present application, regardless of the quality of the original image, an ideal positioning result can be obtained. For the box image shown in the 1040 part, the distortion of the box image itself is relatively serious during acquisition, while there are multiple ventilation hood areas in the box image shown in the 1070 part. In these situations, the solution provided by the present application can still have an accurate positioning effect on the ventilation hood area.

[0095] Furthermore, the effect comparison and description of the method provided by the present application applied to the business process of container positioning are further combined with Fig. 11(a) and Fig. 11(b). Among them, Fig. 11(a) is the original business process of positioning the ventilation hood area of the container, and Fig. 11(b) is the business process of positioning the ventilation hood area of the container using the method provided by the present application. By comparison, it can be seen that different from the original business process, the present application combines computer vision technology in the ventilation hood area positioning, and can also be assisted by less manual review on this basis. Moreover, the effect diagrams of the corresponding positioning results also show that the business process of positioning the ventilation hood area of the container using the method provided by the present application has a faster and better positioning effect compared with the original business process.

[0096] The method provided by the present application can achieve imaging and ventilation hood positioning of the container simultaneously without a vehicle when the container is transported into the yard, greatly reducing the time cost. This method can not only be applied to the ventilation hood positioning of yard containers, but also can detect the overall condition of the ventilation hood of newly produced containers to ensure the good quality of the container ventilation hood. In addition, in the container deformation detection task, applying ventilation hood positioning can greatly improve the accuracy of deformation detection. And this method also has the advantages of high robustness, fast positioning speed and high precision, and can greatly reduce the manual workload.

[0097] It should be understood that although Figures 2 to 11(b) the steps in the flowchart Figures 2 to 11(b)At least a part of the steps therein may include multiple steps or multiple stages. These steps or stages are not necessarily executed and completed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0098] In one embodiment, as Figure 12 shown, a device for determining the air-permeable cover area of a box body is provided. This device can be a software module, a hardware module, or a combination of both to form a part of a computer device. The device 1200 specifically includes:

[0099] An image acquisition module 1201, configured to acquire a box body image and acquire a depth image corresponding to the box body image;

[0100] A candidate determination module 1202, configured to determine multiple air-permeable cover candidate areas in the box body image according to the depth information provided by the depth image;

[0101] A region detection module 1203, configured to input the candidate region images corresponding to each air-permeable cover candidate area into a pre-constructed air-permeable cover area detection network, so that the air-permeable cover area detection network outputs the detection results corresponding to each candidate region image;

[0102] A region determination module 1204, configured to determine the air-permeable cover area of the box body from the multiple air-permeable cover candidate areas according to the detection results.

[0103] In one embodiment, the candidate determination module 1202 is further configured to determine multiple groove areas in the box body image according to the depth information; determine the initial air-permeable cover areas in each groove area based on the depth value distribution of the groove areas; and obtain the multiple air-permeable cover candidate areas from the initial air-permeable cover candidate areas according to the area sizes of the initial air-permeable cover areas.

[0104] In one embodiment, the candidate determination module 1202 is further configured to determine whether there is a depth anomaly area in the groove area based on the depth value distribution of the groove area; the depth anomaly area is an area with a preset depth difference from an adjacent area; the adjacent area is an area adjacent to the depth anomaly area in the groove area; if there is a depth anomaly area in the groove area, the depth anomaly area is used as the initial air-permeable cover area.

[0105] In one embodiment, the candidate determination module 1202 is further configured to use the initial air-permeable cover areas with the area sizes within a preset area size range as the air-permeable cover candidate areas.

[0106] In one embodiment, the area detection module 1203 is further configured to input the candidate area images corresponding to the respective air-permeable cover candidate areas into the air-permeable cover area detection network, so that the air-permeable cover area detection network obtains a first feature map corresponding to the candidate area image through the main path network and a second feature map corresponding to the candidate area image through the branch network, fuse the first feature map and the second feature map to obtain a fused feature map, and output the detection result according to the fused feature map.

[0107] In one embodiment, the air-permeable cover area detection network is further configured to fuse the first feature map and the second feature map according to a first fusion weight corresponding to the first feature map and a second fusion weight corresponding to the second feature map to obtain the fused feature map; wherein, the first fusion weight is greater than the second fusion weight.

[0108] In one embodiment, the above device 1200 may further include: a network construction module, configured to obtain a box body sample image and a depth sample image corresponding to the box body sample image; obtain a positive sample area image corresponding to the air-permeable cover area from the box body sample image and the depth sample image, and obtain a negative sample area image from the groove area of the box body sample image and the depth sample image; use a preset proportion of the positive sample area images and the negative sample area images to train the air-permeable cover area detection network to be trained, and construct the air-permeable cover area detection network.

[0109] For the specific limitations on the box body air-permeable cover area determination device, reference may be made to the limitations on the box body air-permeable cover area determination method in the foregoing, which will not be elaborated here. Each module in the above box body air-permeable cover area determination device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so as to facilitate the processor to call and execute the operations corresponding to the above respective modules.

[0110] In one embodiment, a computer device is provided. The computer device can be a terminal or a server, and its internal structure diagram can be as Figure 13 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with external devices through a network connection. When the computer program is executed by the processor, it implements a method for determining the box body air-permeable cover area.

[0111] Those skilled in the art can understand that Figure 13 the structure shown in Figure 13 is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0112] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0113] In one embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0114] In one embodiment, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in the above method embodiments.

[0115] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it may include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in this application may include at least one of non-volatile and volatile memories. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0116] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0117] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A method for determining the air-permeable cover area of a box body, characterized in that, The method includes: Obtaining a box image and obtaining a depth image corresponding to the box image; If there is a depth anomaly region among multiple groove regions in the box image, then taking the depth anomaly region as a preliminary selection region for the ventilation hood; obtaining multiple ventilation hood candidate regions in the box image from the preliminary selection regions for the ventilation hood according to the region sizes of the preliminary selection regions for the ventilation hood; the multiple groove regions are determined according to the depth information provided by the depth image; the depth anomaly region is a region with a preset depth difference from an adjacent region; the adjacent region is a region adjacent to the depth anomaly region in the groove region; the ventilation hood candidate region belongs to a suspected ventilation hood region; Inputting the candidate region image corresponding to each ventilation hood candidate region into a pre-constructed ventilation hood region detection network, so that the ventilation hood region detection network outputs a detection result corresponding to each candidate region image; the detection result is used to indicate whether the corresponding candidate region image belongs to a ventilation hood; Determining the ventilation hood region of the box from the multiple ventilation hood candidate regions according to the detection result.

2. The method according to claim 1, wherein Before the step of if there is a depth anomaly region among multiple groove regions in the box image, then taking the depth anomaly region as a preliminary selection region for the ventilation hood, it further includes: Based on the depth value distribution of the groove region, determining whether there is a depth anomaly region in the groove region.

3. The method according to claim 1, wherein The step of obtaining multiple ventilation hood candidate regions in the box image from the preliminary selection regions for the ventilation hood according to the region sizes of the preliminary selection regions for the ventilation hood includes: Taking the preliminary selection region for the ventilation hood with the region size within a preset region size range as the ventilation hood candidate region.

4. The method according to claim 1, characterized in that, The step of inputting the candidate region image corresponding to each ventilation hood candidate region into a pre-constructed ventilation hood region detection network, so that the ventilation hood region detection network outputs a detection result corresponding to each candidate region image, includes: Inputting the candidate region image corresponding to each ventilation hood candidate region into the ventilation hood region detection network, so that the ventilation hood region detection network obtains a first feature map corresponding to the candidate region image through the main path network and obtains a second feature map corresponding to the candidate region image through the branch network, fusing the first feature map and the second feature map to obtain a fused feature map, and outputting the detection result according to the fused feature map.

5. The method according to claim 4, characterized in that The ventilation hood region detection network is further used to fuse the first feature map and the second feature map to obtain the fused feature map according to a first fusion weight corresponding to the first feature map and a second fusion weight corresponding to the second feature map; wherein, the first fusion weight is greater than the second fusion weight.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtaining a box sample image and obtaining a depth sample image corresponding to the box sample image; Obtaining a positive sample region image corresponding to the ventilation hood region from the box sample image and the depth sample image, and obtaining a negative sample region image from the groove regions of the box sample image and the depth sample image; Train the breathable hood area detection network to be trained using the positive sample area images and the negative sample area images with a preset ratio, and construct the breathable hood area detection network.

7. A device for determining the air-permeable cover area of a box body, characterized in that, The device includes: An image acquisition module, configured to acquire a box image and a depth image corresponding to the box image; A candidate determination module, configured to, if there is a depth anomaly area in multiple groove areas in the box image, use the depth anomaly area as a primary selection area for the breathable hood; according to the area sizes of the primary selection areas for the breathable hood, obtain multiple candidate areas for the breathable hood in the box image from the primary selection areas for the breathable hood; the multiple groove areas are determined according to the depth information provided by the depth image; the depth anomaly area is an area with a preset depth difference from an adjacent area; the adjacent area is an area adjacent to the depth anomaly area in the groove area; the candidate areas for the breathable hood belong to suspected breathable hood areas; An area detection module, configured to input the candidate area images corresponding to the candidate areas for the breathable hood into a pre-constructed breathable hood area detection network, so that the breathable hood area detection network outputs detection results corresponding to the candidate area images; the detection results are used to indicate whether the corresponding candidate area images belong to the breathable hood; An area determination module, configured to determine the breathable hood area of the box from the multiple candidate areas for the breathable hood according to the detection results.

8. The device according to claim 7, characterized in that, The candidate determination module is further configured to determine whether there is a depth anomaly area in the groove area based on the depth value distribution of the groove area.

9. The device according to claim 7, characterized in that, The candidate determination module is configured to use the primary selection areas for the breathable hood with the area sizes within a preset area size range as the candidate areas for the breathable hood.

10. The device according to claim 7, characterized in that, The area detection module is configured to input the candidate area images corresponding to the candidate areas for the breathable hood into the breathable hood area detection network, so that the breathable hood area detection network obtains a first feature map corresponding to the candidate area image through the main path network and a second feature map corresponding to the candidate area image through the branch network, fuse the first feature map and the second feature map to obtain a fused feature map, and output the detection results according to the fused feature map.

11. The device according to claim 10, characterized in that, The breathable hood area detection network is further configured to fuse the first feature map and the second feature map to obtain the fused feature map according to a first fusion weight corresponding to the first feature map and a second fusion weight corresponding to the second feature map; wherein, the first fusion weight is greater than the second fusion weight.

12. The device according to any one of claims 7 to 11, characterized in that, The device further includes: a network construction module, configured to acquire a box sample image and a depth sample image corresponding to the box sample image; obtain a positive sample area image corresponding to the breathable hood area from the box sample image and the depth sample image, and obtain a negative sample area image from the groove areas of the box sample image and the depth sample image; train the breathable hood area detection network to be trained using a preset ratio of the positive sample area images and the negative sample area images, and construct the breathable hood area detection network.

13. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

14. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images