Warehouse cargo monitoring identification method based on image processing technology

Abstract

The invention relates to a warehouse cargo monitoring identification method based on an image processing technology, and belongs to the technical field of image processing, and the method comprises the steps: obtaining a monitoring instruction, shooting an identification image, delimiting light and shade boundaries from the identification image, carrying out the light and shade equalization processing of an area which is divided by each light and shade boundary in the identification image, defining an identification frame in the identification image after light and shade equalization processing; taking a regional image in the recognition frame in the recognition image as a recognition object, inputting the recognition object into a pre-constructed recognition model, and outputting a recognition result through the recognition model; wherein the identification result at least comprises position information of the identification frame, a judgment result of whether goods exist in a warehouse area corresponding to the identification frame, and a goods type when the goods exist in the area corresponding to the identification frame; training images and verification images are obtained through regular collection, and the recognition model is optimized through the training images and the verification images. The method has the effect of improving the recognition reliability of the image processing technology on the warehouse goods.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a warehouse cargo monitoring and identification method based on image processing technology. Background Art

[0002] With the rapid development of economy and logistics industry, warehouse cargo management has become an important issue. For large warehouses, how to quickly and accurately classify and monitor cargo in the case of large quantity and rich variety of cargo has become an important task of warehouse management.

[0003] Traditional warehouse management methods mainly rely on manual operations, but this method is inefficient, time-consuming, and prone to human errors. For this reason, image processing technology has been widely used in warehouse management in recent years: by installing high-definition cameras in the warehouse, and then using the cameras to capture images in the warehouse, and then using image processing technology to identify and process them to determine whether there are goods stored in the warehouse, features can also be extracted from the image and compared with the reference image stored in the database in advance to determine the type of goods, where the reference image here can be considered as images of different types of warehouse goods.

[0004] However, in practical applications, the image of the goods captured may be inconsistent with the reference image due to the influence of warehouse lighting, which may result in failure to successfully match the reference image when determining the type of goods, thus affecting recognition reliability. This needs to be improved. Summary of the invention

[0005] In order to improve the influence of warehouse light on the goods during the identification process and improve the reliability of image processing technology in identifying warehouse goods, the present application provides a warehouse goods monitoring and identification method based on image processing technology.

[0006] In the first aspect, the present application provides a warehouse cargo monitoring and identification method based on image processing technology, which adopts the following technical solution: Obtaining a monitoring instruction, capturing a recognition image, delineating a light-dark dividing line from the recognition image, performing a light-dark balancing process on an area in the recognition image that is divided by each light-dark dividing line, and delineating a recognition frame in the recognition image after the light-dark balancing process; The area image in the recognition frame in the recognition image is used as a recognition object, the recognition object is input into a pre-built recognition model, and a recognition result is output through the recognition model; wherein the recognition result at least includes the position information of the recognition frame, the determination result of whether there are goods in the warehouse area corresponding to the recognition frame, and the type of goods when there are goods in the area corresponding to the recognition frame; Training images and verification images are regularly collected, and the recognition model is optimized using the training images and verification images.

[0007] By adopting the above technical solution, the light-dark dividing line refers to the light and shadow lines generated in the recording area of ​​the camera in the warehouse under the influence of the light source. The light-dark dividing line divides the recording area into several areas. If the same goods are in the aforementioned different areas, the goods image taken by the camera for the goods and the pre-stored reference image are likely to be inconsistent (such as inconsistent brightness), which is not conducive to the successful recognition of the goods by the recognition model; or when the goods are in two areas (hereinafter referred to as area A and area B) divided by a certain light-dark dividing line at the same time, that is, one part of the goods is located in area A and the other part is located in area B, then the goods image taken by the camera for the goods is likely to have uneven brightness. Therefore, based on the above two situations, the present application proposes to pre-balance the brightness of the recognition image before using the recognition model for image recognition. The equalization processing here can be understood as a processing operation used to weaken the influence of light and dark light on the captured goods image and to improve the recognition success rate of the recognition model for the goods image, so as to ultimately ensure the reliability of the recognition model for image processing and recognition.

[0008] Optionally, performing light-dark balance processing on the area divided by each light-dark dividing line in the recognition image, and defining a recognition frame in the recognition image after the light-dark balance processing, comprises: The area formed by the light-dark dividing line in the recognition image is used as the recognition area, and a balanced block is selected from each of the recognition areas; Using the pre-built brightness balance model, perform brightness balance on the image fragments corresponding to each balanced block to obtain a deduced image, compare the deduced image with the reference image in the pre-stored image library, and if there is a reference image containing the deduced image, then the balance is completed, and the deduced image is used to replace the image corresponding to the corresponding balanced block in the identified image, and the goods corresponding to the reference image are used as the associated goods of the balanced block; Based on the associated goods, all balanced blocks are fused and delineated to form an identification frame.

[0009] By adopting the above technical solution, an equalization block is cut out from each recognition area formed after the recognition image is divided by the light and dark dividing line, and the brightness of the image fragments corresponding to the equalization block is equalized. Through the sensitive equalization processing operation, on the one hand, it is used to balance the brightness of the entire recognition image, and on the other hand, it can make the equalized recognition image match the reference image to ensure the recognition accuracy and reliability of the subsequent recognition model for the recognition image.

[0010] Optionally, the pre-built brightness equalization model is used to perform brightness equalization on the image fragments corresponding to each equalization block to obtain a deduced image, and the deduced image is compared with a reference image in a pre-stored image library. If there is a reference image containing the deduced image, completing the equalization includes: Determine a first equalization value according to the position information of each equalization block in the identified image, and for each equalization block, perform shallow equalization processing on the image fragments corresponding to the equalization block through a pre-constructed brightness equalization model according to the corresponding first equalization parameter to obtain a deduced image; According to the position information of each equalization block in the identified image, a second equalization value corresponding to the equalization block is determined; for each equalization block, according to the corresponding second equalization parameter, a deep equalization process is performed on the corresponding deduced image through the brightness equalization model, and the deduced image after each deep equalization process is compared with a reference image in a pre-stored image library; If there is a reference image that includes a deduced image after deep equalization processing, the deep equalization processing of the corresponding equalization block is completed; if not, the second equalization parameter corresponding to the equalization block is re-determined, and the corresponding deduced image is deep-equalized again through the brightness equalization model until there is a reference image that includes a deduced image after deep equalization processing.

[0011] By adopting the above technical solution, this solution proposes two levels of equalization operations for the image fragments corresponding to the equalization block, the first level is a shallow equalization processing operation, and the second level is a deep equalization processing operation. Among them, the shallow equalization processing operation is to balance the uneven brightness of the cargo image (i.e., the recognition image) caused by the unevenness of light exposure in different recognition areas; and considering that the cargo will produce shadows when illuminated by the light source, if the shadow covers other cargo, then the cargo image will appear uneven in brightness due to the shadow. Therefore, the deep-level equalization processing operation of this application is used to equalize the deduced image after the shallow equalization processing, that is, to improve the uneven brightness of the cargo image caused by the shadow that may be cast.

[0012] Optionally, determining the second equalization parameter corresponding to the equalization block includes: According to the pre-stored cargo attributes and the position information of the balancing block in the recognition image, the shadow projection degrees of all preset cargoes in the recognition area where the balancing block is located are determined, a second balancing data set is determined for the balancing block, and a deep balancing algorithm is generated for the balancing block based on the second balancing data set; wherein the cargo attributes at least include the light transmittance of the cargo outer packaging material, and the second balancing data set refers to a data set consisting of a number of shadow projection degrees; The deep equalization processing of the corresponding deduced image by using the brightness equalization model includes: The deep equalization algorithm is loaded through the brightness equalization model to perform deep equalization processing on the corresponding deduced image.

[0013] By adopting the above technical solution, it can be known from the foregoing that the goods are prone to produce shadows in the identification area due to the influence of light source illumination, and due to the different light source intensities in different identification areas, the projection depths of the goods in different identification areas are correspondingly different (that is, different shadow projection degrees), and further, based on the properties of the goods, it can be known that the light transmittance of goods with outer packaging of different materials is different, so the shadow projection degrees of goods with different light transmittances in the same identification area will also be different. For this reason, this solution takes the above two factors into consideration and determines the corresponding second balanced data set for each balanced block. It can be considered that the second balanced data set contains all shadow projection degrees of different goods in the identification area where the current balanced block is located, and this The application will generate a deep equalization algorithm for the corresponding equalization block based on the second equalization data set, and the deep equalization algorithm can be considered as an algorithm composed of selecting any one or more shadow addition degrees from the second equalization data set, so as to simulate the image fragments as images of goods covered by one or more shadows. The process of loading the deep equalization algorithm with the brightness equalization model is to adjust the brightness of the image fragments to simulate the process of removing the goods covered by the aforementioned multiple shadows. Finally, the deduced image after the deep equalization processing is completed can be considered as the image taken when the goods are not covered by shadows, and also refers to the image that can be successfully matched with the reference image, so as to improve the recognition reliability and accuracy of the subsequent recognition model.

[0014] Optionally, based on the associated goods, all the balancing blocks are merged and delineated to form an identification frame, including: Determine the coverage area of ​​the associated goods corresponding to each balancing block in the identification image according to the size of the associated goods corresponding to each balancing block and the position information of each balancing block in the identification image, and define an identification frame for each coverage area according to the boundary line of each coverage area; If there are several mutually overlapping first coverage areas, the identification frames of the first coverage areas are merged to form a unique identification frame; If there are several second coverage areas that overlap and do not overlap with each other, it is considered that the associated goods corresponding to the second coverage areas are stacked on each other, and the upper and lower stacking order of the stacked associated goods is determined, and the display form of the identification box corresponding to each second coverage area is adjusted in sequence to determine the upper and lower stacking order corresponding to the second coverage area through the display form of the identification box.

[0015] By adopting the above technical solution, this solution proposes to limit the recognition frame before using the recognition model for recognition, so as to use the number of recognition frames to represent the approximate number of goods and use the position of the recognition frame to represent the placement of the goods in the warehouse.

[0016] Optionally, the method further includes: When there are several second coverage areas that overlap and do not overlap with each other, the stacking stability is determined and output based on the overlapping parts; if the stacking stability does not meet the preset stability requirement, a preset mark is added to the identification frame of the corresponding second coverage area, and the preset mark is used to indicate that there are goods under the associated goods corresponding to the corresponding identification frame that are completely obscured by the associated goods and cannot be identified and displayed in the identification image.

[0017] By adopting the above technical solution, considering that the camera has a single shooting angle, for goods stacked together, it is easy for some goods to be blocked by other goods and cannot be displayed in the recognition image. The present application specifically proposes: when there are related goods stacked on each other, by judging the stacking stability, it is determined whether there are any goods in the stacked goods that are completely blocked and cannot be recognized in the recognition image, so as to improve the comprehensiveness and accuracy of the recognition of the quantity of goods.

[0018] Optionally, selecting a balanced block from each of the identified areas includes: Selecting a minimum size from all preset cargo sizes, and dividing each identification area into a number of screening blocks according to the minimum size, wherein the size of each screening block is consistent with the minimum size; A balancing block is selected in each screening block, and the condition is that each screening block contains at least two or more balancing blocks.

[0019] By adopting the above-mentioned technical solution, in order to improve the success rate of matching the image fragments corresponding to the balanced block after the balanced processing and the reference image, the present application proposes the above-mentioned selection scheme for selecting the balanced block, thereby reducing the situation where multiple local parts of the goods appear in the image fragments corresponding to a single balanced block.

[0020] In a second aspect, the present application provides a warehouse cargo monitoring and identification system based on image processing technology, including an identification frame delineation module, which is used to obtain monitoring instructions, capture an identification image, delineate light and dark dividing lines from the identification image, perform light and dark balancing processing on the area in the identification image divided by each light and dark dividing line, and delineate an identification frame in the identification image after the light and dark balancing processing; A cargo identification module, used to take the area image in the identification image and the identification frame as the identification object, input the identification object into a pre-built identification model, and output the identification result through the identification model; wherein the identification result at least includes the position information of the identification frame, the determination result of whether there are cargoes in the warehouse area corresponding to the identification frame, and the cargo type when there are cargoes in the area corresponding to the identification frame; The model optimization module is used to regularly collect training images and verification images, and optimize the recognition model using the training images and verification images.

[0021] In a third aspect, the present application provides a warehouse cargo monitoring and identification device based on image processing technology, including a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute any one of the methods described in the first aspect.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any one of the methods described in the first aspect.

[0023] In summary, this application includes the following beneficial technical effects: This application performs feature extraction and analysis on the images taken by the camera in the warehouse based on the Yolov5 algorithm, and then identifies whether there are goods stored in the warehouse and the types of goods stored, thereby improving the recognition efficiency; and before recognition, this application also performs preprocessing operations on the images taken by the camera to achieve light and dark balance processing of the image, improve the impact of uneven brightness caused by the light source on the captured image, and improve recognition reliability and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 It is a flow chart of a warehouse cargo monitoring and identification method based on image processing technology disclosed in an embodiment of the present application.

[0026] Figure 2 It is a structural block diagram of a warehouse cargo monitoring and identification system based on image processing technology disclosed in an embodiment of the present application.

[0027] Explanation of the accompanying drawings: 1. Identification frame delineation module; 2. Goods identification module; 3. Model optimization module. DETAILED DESCRIPTION

[0028] The embodiment of the present application discloses a warehouse cargo monitoring and identification method based on image processing technology (hereinafter referred to as the monitoring and identification method). The execution subject of the monitoring and identification method is a warehouse cargo monitoring and identification system based on image processing technology (hereinafter referred to as the monitoring and identification system). Figure 1 The specific steps of the monitoring identification method are elaborated in detail.

[0029] S101, obtaining a monitoring instruction, capturing a recognition image, drawing a light-dark dividing line from the recognition image, performing a light-dark balancing process on the area in the recognition image divided by each light-dark dividing line, and drawing a recognition frame in the recognition image after the light-dark balancing process.

[0030] In implementation, illustratively, the monitoring instruction can be triggered manually or automatically by the monitoring identification system at specified frequency or duration. The monitoring identification instruction is pre-communicated with a camera, and the camera is pre-installed in the warehouse. The number of cameras can be manually increased according to the number of warehouses, and the cameras are distinguished by pre-set identification information (such as camera ID and / or account password). This application assumes that the recording area of ​​the camera covers the area in the warehouse for placing goods. Whenever a monitoring instruction is received, the monitoring identification system is used to obtain the video stream recorded by the camera. The recognition image can be considered as a single frame image captured by the monitoring identification system from the video stream. The recognition image is used to represent the picture inside the warehouse taken from the camera angle.

[0031] After obtaining the recognition image, the monitoring recognition system is used to add a mask to the recognition image, and use the area image with the mask added to replace the recognition image as a new recognition image, and the mask can be added at several positions, that is, multiple masks can be added to a single recognition image, and finally a number of recognition images consistent with the number of masks added are split out. It should be mentioned that the mask here refers to the mask in Python, that is, a method for processing an image, which is used to shield the pixel values ​​of a specific area or set them to a specific value. Since it is a prior art, it will not be described here.

[0032] In addition, the area where the mask is added in the recognition image can be a manually preset area, or it can be a masked area based on the warehouse's recording area and the location where goods have been stored in the historical period; that is, the area within the recording area where goods may be placed is retained.

[0033] Based on the recognition image obtained by the above processing, the monitoring recognition system will draw a light-dark dividing line in the recognition image. The light-dark dividing line of this application refers to the light and shadow lines generated in the recording area of ​​the warehouse camera under the influence of the light source. For example, for a camera installed on the top wall of the warehouse, its recording area is the area of ​​the warehouse floor. During the light source irradiation, an illuminated area will be left on the ground. There is a boundary line between the illuminated area and the unilluminated area, that is, the light and shadow line. The light source mentioned in the embodiment of this application refers to a fixed light source (i.e., a light source with a fixed illumination angle), such as a lamp installed in a warehouse, and the embodiment of this application assumes that the installation position and recording area of ​​the camera relative to the warehouse are fixed, so the number and position of the light and shadow lines (i.e., light-dark dividing lines) generated by the light source in the recording area are fixed, and the monitoring recognition system is assumed to have pre-recorded the position of the light and dark dividing line relative to the inner wall of the warehouse, so the monitoring recognition system can draw a light-dark dividing line in the recognition image based on the aforementioned pre-stored position information and the corresponding position relationship between the recognition image and the recording area in the warehouse.

[0034] Combined with the above, it can be known that the light-dark dividing line refers to the dividing line between two areas where the light and dark brightness are inconsistent due to the illumination of the light source. Therefore, this application will divide the recognition image into several areas based on the light-dark dividing line, and the area boundary line is the light-dark dividing line and / or the mask boundary line. Since different areas are illuminated by the light source and the light and dark brightness are inconsistent, the corresponding manifestation also leads to uneven brightness in the recognition image. If the goods are in the aforementioned different areas, or different parts of the goods are respectively located in the aforementioned different areas, then the goods will also be affected by the light source, resulting in uneven brightness in the goods image captured by the camera. If the goods image captured in this case is included in the recognition image, it will be detrimental to the subsequent accurate recognition of the goods. For this reason, this application proposes to perform light and dark equalization processing on the image fragments corresponding to the areas with inconsistent light and dark brightness in the recognition image. In short, it is to adjust the image brightness value to reduce the influence of the light source. After completing the light and dark equalization processing, a recognition frame is delineated in the recognition image so that the goods recognition operation can be performed on the image framed in the recognition frame.

[0035] Specifically, the specific steps of light and dark equalization processing are as follows: S1011, taking the region formed by the light-dark dividing line in the recognition image as the recognition region, and selecting a balanced block from each recognition region; S1012, using a pre-built brightness balance model, perform brightness balance on the image fragments corresponding to each balanced block to obtain a deduced image, compare the deduced image with a reference image in a pre-stored image library, and if there is a reference image that contains the deduced image, then the balance is completed, and the image corresponding to the corresponding balanced block in the recognition image is replaced with the deduced image, and the goods corresponding to the reference image are used as the associated goods of the balanced block; S1013, based on the associated goods, all the balanced blocks are merged and delineated to form an identification frame.

[0036] The step of “selecting a balanced block from each identified area” in S1011 includes the following sub-steps: Select the minimum size from all preset cargo sizes, and divide each identification area into several screening blocks according to the minimum size, and the size of each screening block is consistent with the minimum size; A balancing block is selected in each screening block, and the following conditions are satisfied: each screening block contains at least two or more balancing blocks.

[0037] In S1012, “using a pre-built brightness equalization model to perform brightness equalization on the image fragments corresponding to each equalization block to obtain a deduced image, and comparing the deduced image with a reference image in a pre-stored image library, and completing the equalization if there is a reference image that includes the deduced image” includes the following sub-steps: According to the position information of each equalization block in the recognition image, a first equalization value is determined, and for each equalization block, according to the corresponding first equalization parameter, a pre-constructed brightness equalization model is used to perform shallow equalization processing on the image fragments corresponding to the equalization block to obtain a deduced image; According to the position information of each balancing block in the recognition image, according to the pre-stored cargo attributes, and the position information of the balancing block in the recognition image, the shadow projection degrees of all preset cargoes in the recognition area where the balancing block is located are determined, a second balancing data set is determined for the balancing block, and a deep balancing algorithm is generated for the balancing block based on the second balancing data set; wherein the cargo attributes at least include the light transmittance of the cargo outer packaging material, and the second balancing data set refers to a data set consisting of a number of shadow projection degrees; For each equalization block, according to the corresponding second equalization parameter, the corresponding deep equalization algorithm is loaded through the brightness equalization model to perform deep equalization processing on the corresponding deduced image, and the deduced image after each deep equalization processing is compared with the reference image in the pre-stored image library; If there is a reference image that contains a deduced image after deep equalization processing, the deep equalization processing of the corresponding equalization block is completed; if not, the deep equalization algorithm corresponding to the equalization block is re-determined, and the corresponding deduced image is deep-equalized again through the brightness equalization model until there is a reference image that contains a deduced image after deep equalization processing.

[0038] In implementation, the monitoring and identification system is used to define the area in the identification image that is formed by the light and dark dividing line as the identification area. The monitoring and identification system pre-stores the cargo attributes of all preset cargoes. The preset cargoes here refer to the cargoes that may be stored in the warehouse. The cargo attributes include cargo name, cargo size (here the size can be length*width*height), light transmittance of the cargo outer packaging material, and reference image. The reference image here refers to the cargo image captured by the camera under the illumination of the specified light source. The specified light source here satisfies: when the cargo is placed at any position in the recording area, the light intensity after being illuminated by the specified light source is the same, that is, the brightness of the cargo image captured by the cargo at any position in the recording area will not be affected by the illumination angle and light intensity of the light source.

[0039] The monitoring and identification system will find the smallest cargo from all pre-stored preset cargoes, and determine the minimum size based on its corresponding size. Since the camera in the embodiment of the present application is installed on the top wall of the warehouse by default and the camera viewing angle is a top-down view, the minimum size here is specifically expressed as length*width.

[0040] The monitoring and identification system is also used to divide each identification area into a number of screening blocks of the same size as the minimum size, and select a balanced block from each screening block, and satisfy: the number of balanced blocks selected from each screening block is at least two.

[0041] Next, the monitoring and identification system is used to determine the first balancing value according to the position information of each balancing block in the identification image (which can be expressed in the form of coordinates). Specifically, the monitoring and identification system pre-stores the first balancing value corresponding to the reference area and each reference area. The reference area here refers to the area divided by the light-dark dividing line in the image obtained by the camera shooting the recording area. Since different reference areas have different illumination intensities when illuminated by the light source, the brightness reflected in the image after being photographed by the camera is also different. Therefore, the first balancing value is used to characterize the brightness here to achieve the distinction between different reference areas, and the brightness balancing model is used to determine the reference area and the first balancing value corresponding to the corresponding identification area according to the identification area where the balancing block is located, and then adjust the brightness of the image fragments corresponding to the balancing block (i.e., shallow balancing processing), and the adjusted value is consistent with the first balancing value. In addition, the first balancing value can be expressed as a positive number or a negative number. A negative number indicates that the brightness of the image fragments corresponding to the balancing block is lowered, and a corresponding positive number indicates that it is increased.

[0042] The recognition image after the above shallow equalization processing is the deduced image. On this basis, the monitoring recognition system will determine the second equalization data set for each equalization block according to the correspondence table, wherein the second equalization data set is composed of a number of shadow projections, and the shadow projection refers to the brightness change value caused by the shadow (i.e., projection) generated by the goods receiving light source illumination in the recognition area covering other goods or the inner wall of the warehouse, on the brightness of the image taken by the camera. Correspondingly, the above correspondence table includes three fields: light transmittance of the outer packaging material of the goods, reference area, and shadow projection, which are used to reflect the shadow projection generated by the goods with different light transmittance in different reference areas under the influence of light sources.

[0043] Therefore, when the monitoring and identification system determines the second balanced data set for each balanced block, it will find the shadow projection degree corresponding to the light transmittance of all goods outer packaging materials corresponding to the reference area in which the target balanced block is located from the correspondence table, thereby generating the second balanced data set.

[0044] Next, the monitoring and identification system will generate a deep balancing algorithm for the balancing block based on the second balancing data set. The deep balancing algorithm can be understood as a formula for the weighted sum of all shadow projections contained in the second balancing data set, such as: Shadow projection i *A i ; Where 0<i≤N, N represents the total number of shadow addition degrees contained in the second balanced data set, and the shadow addition degree i A refers to the i-th shadow addition degree in the second balanced data set. irefers to the weight value corresponding to the i-th shadow addition degree in the second balanced data set, and A i ≤0, that is, A i The value is 0 or an integer less than 0.

[0045] By adjusting the A corresponding to the shadow addition degree i The deep equalization algorithm is adjusted by the value of , and after each adjustment of the deep equalization algorithm, the adjusted deep equalization algorithm is loaded using the brightness equalization model to perform deep equalization processing on the deduced image of the corresponding equalization block. Specifically, the deep equalization processing here refers to further adjusting the brightness value of the deduced image, wherein,

[0046] After each deep equalization process is completed, the monitoring and identification system is used to compare the deduced image after deep equalization process with all the reference images in the pre-stored image library one by one to determine whether the deduced image is included in any of the reference images in the image library, that is, whether the deduced image is a part of any of the reference images in the image library. If so, it is considered that the comparison is consistent, and the adjustment of the deep equalization algorithm is stopped, and finally the deep equalization process of the balancing block is completed, and the current deep equalization algorithm is used as the final deep equalization algorithm of the balancing block, and the goods corresponding to the reference image that is consistent with the comparison are used as the associated goods of the balancing block. If not, the deep equalization algorithm of the balancing block is readjusted according to the above method, and the deduced image of the balancing block is deep-equalized again through the brightness balance model based on the adjusted deep equalization algorithm until a reference image that is consistent with the comparison is found for the deduced image.

[0047] Wherein, S1013 specifically includes the following sub-steps: According to the size of the associated goods corresponding to each balanced block and the position information of each balanced block in the recognition image, the coverage area of ​​the associated goods corresponding to each balanced block in the recognition image is determined, and according to the boundary line of each coverage area, an recognition frame is drawn for each coverage area; If there are several mutually overlapping first coverage areas, the identification frames of the first coverage areas are merged to form a unique identification frame; If there are several second coverage areas that overlap and do not overlap with each other, it is considered that the associated goods corresponding to the second coverage areas are stacked on each other, and the upper and lower stacking order of the stacked associated goods is determined, and the display form of the identification box corresponding to each second coverage area is adjusted in sequence to determine the upper and lower stacking order corresponding to the second coverage area through the display form of the identification box.

[0048] In implementation, after the associated goods corresponding to the balancing block are known, the corresponding size is obtained according to the goods attributes of the corresponding goods, and the position information and coverage area of ​​the corresponding goods in the recognition image are determined based on the size. The coverage area here refers to the area corresponding to the goods of the corresponding size when they are completely photographed by the camera and displayed in the recognition image. The coverage area can be expressed in the form of a coordinate interval or a coordinate set, and the boundary frame line of the recognition frame described in this application is the boundary line of the coverage area, and the area enclosed by the recognition frame is the coverage area.

[0049] In addition, if there are several overlapping coverage areas (i.e., first coverage areas), the associated goods represented by the first coverage areas are considered to be the same goods. At this time, the monitoring and identification system merges the identification frames corresponding to all the aforementioned first coverage areas into a unique identification frame to achieve deduplication processing.

[0050] In addition, if there are several coverage areas that do not overlap but have overlapping parts (i.e., the second coverage area), and since the default shooting angle of the camera in this application is a top-down shooting, this situation is considered to be that there are several goods stacked on each other along the height direction of the warehouse. At this time, the monitoring and identification system will determine the upper and lower stacking order of the associated goods corresponding to the second coverage area, that is, the upper and lower position relationship in the height direction, and the determination method can be: according to the final deep balancing algorithm corresponding to the balancing block corresponding to the second coverage area, the absolute values ​​of all weight values ​​in the corresponding deep balancing algorithm are summed to obtain the total value, and the order of the total values ​​from small to large is used as the stacking order from top to bottom.

[0051] After determining the up-and-down stacking order, the display frames of all first covering areas except the first covering area corresponding to the topmost associated goods (hereinafter referred to as the display frames to be adjusted) are adjusted, specifically for adjusting the display form of the display frames to be adjusted, such as dividing the frame line corresponding to the target identification frame to be adjusted into a solid line part and a dotted line part, the dotted line part refers to the frame line in the area corresponding to the overlapping part, and the overlapping part here refers to the overlapping part between the first covering area corresponding to the target display frame to be adjusted and the first covering area above it in the up-and-down stacking order, and the target display frame to be adjusted refers to any display frame to be adjusted.

[0052] Optionally, the monitoring and identification method further includes: When there are several second coverage areas that overlap and do not overlap with each other, the stacking stability is determined and output based on the overlapping parts; if the stacking stability does not meet the preset stability requirement, a preset mark is added to the identification frame of the corresponding second coverage area, and the preset mark is used to indicate that there are goods under the associated goods corresponding to the corresponding identification frame that are completely obscured by the associated goods and cannot be identified and displayed in the identification image.

[0053] In implementation, for related goods that are stacked on each other, the application also proposes to determine their stacking stability, and add a preset mark to the identification box whose stacking stability does not meet the preset stability requirements. The preset mark can be added by adjusting the color of the identification box to a preset color.

[0054] Exemplarily, the stacking stability is determined as follows: for any two overlapping second coverage areas, determine the size of the overlapping area, and find the corresponding minimum overlapping area from the preset reference table according to the associated goods corresponding to the aforementioned second coverage area. If the current overlapping area is smaller than the minimum overlapping area found, it is considered that the preset stability requirement is not met. The reference table contains three fields, namely: upper goods, lower goods, and minimum overlapping area; here, upper goods and lower goods are used to represent two goods stacked on each other, and upper and lower are used to represent the stacking order.

[0055] S102, taking the area image in the recognition image and within the recognition frame as the recognition object, inputting the recognition object into a pre-built recognition model, and outputting a recognition result through the recognition model; wherein the recognition result at least includes the position information of the recognition frame, the determination result of whether there are goods in the warehouse area corresponding to the recognition frame, and the type of goods when there are goods in the area corresponding to the recognition frame; S103, regularly collecting training images and verification images, and optimizing the recognition model using the training images and verification images.

[0056] In implementation, after determining the recognition frame, the image of the area framed by the recognition frame is used as the recognition object, and the recognition object is input into a pre-built recognition model. The recognition model disclosed in the embodiment of the present application is used to identify the recognition object based on a preset image recognition algorithm and output the recognition result. Exemplarily, the image recognition algorithm mentioned here can be the yolov5 algorithm, which extracts image features and identifies whether there are goods and the type of goods. Since the yolov5 algorithm is a prior art, its principle will not be elaborated on here.

[0057] In addition, the present application also proposes to regularly generate a basic image set, define a label for each reference image in the basic image set, and classify all basic images into two categories: training images and verification images through labels, train the recognition model through training images, and verify the recognition accuracy of the recognition model through verification images. The basic images in the basic image set here can be images obtained by taking pictures of the recording area by cameras in the warehouse.

[0058] The present application also discloses a warehouse cargo monitoring and identification system based on image processing technology. Figure 2 ,include: The recognition frame delineation module 1 is used to obtain monitoring instructions, capture recognition images, delineate light and dark dividing lines from the recognition images, perform light and dark balancing processing on the areas in the recognition images divided by each light and dark dividing line, and delineate recognition frames in the recognition images after the light and dark balancing processing; The cargo identification module 2 is used to take the area image in the identification image and the area within the identification frame as the identification object, input the identification object into the pre-built identification model, and output the identification result through the identification model; wherein the identification result at least includes the position information of the identification frame, the determination result of whether there is cargo in the warehouse area corresponding to the identification frame, and the cargo type when there is cargo in the area corresponding to the identification frame; The model optimization module 3 is used to regularly collect training images and verification images, and optimize the recognition model using the training images and verification images.

[0059] Optionally, the recognition frame delineation module 1 is also used to use the area formed by the light and dark dividing lines in the recognition image as the recognition area, and select the balancing block from each recognition area; it is also used to use a pre-built brightness balancing model to perform brightness balancing on the image fragments corresponding to each balancing block to obtain a deduced image, and compare the deduced image with a reference image in a pre-stored image library. If there is a reference image containing the deduced image, the balancing is completed, and the deduced image is used to replace the image corresponding to the corresponding balancing block in the recognition image, and the goods corresponding to the reference image are used as the associated goods of the balancing block; it is also used to fuse all the balancing blocks based on the associated goods, and delineate them to form a recognition frame.

[0060] Optionally, the identification frame delineation module 1 is also used to determine a first equalization value according to the position information of each equalization block in the identification image, and for each equalization block, according to the corresponding first equalization parameter, perform shallow equalization processing on the image fragments corresponding to the equalization block through a pre-built brightness equalization model to obtain a deduced image; it is also used to determine a second equalization value corresponding to the equalization block according to the position information of each equalization block in the identification image, and for each equalization block, according to the corresponding second equalization parameter, perform deep equalization processing on the corresponding deduced image through a brightness equalization model, and compare the deduced image after each deep equalization processing with a reference image in a pre-stored image library; it is also used to complete the deep equalization processing of the corresponding equalization block if there is a reference image that includes the deduced image after the deep equalization processing; if not, re-determine the second equalization parameter corresponding to the equalization block, and again perform deep equalization processing on the corresponding deduced image through the brightness equalization model until there is a reference image that includes the deduced image after the deep equalization processing.

[0061] Optionally, the identification frame delineation module 1 is also used to determine the shadow projection degrees of all preset goods in the identification area where the balancing block is located according to pre-stored goods attributes and position information of the balancing block in the identification image, determine a second balancing data set for the balancing block, and then generate a deep balancing algorithm for the balancing block based on the second balancing data set; wherein the goods attributes include at least the light transmittance of the outer packaging material of the goods, and the second balancing data set refers to a data set composed of several shadow projection degrees; and is also used to load the deep balancing algorithm through the brightness balancing model to perform deep balancing processing on the corresponding deduced image.

[0062] Optionally, the identification frame delineation module 1 is also used to determine the coverage area of ​​the associated goods corresponding to each balancing block in the identification image according to the size of the associated goods corresponding to each balancing block and the position information of each balancing block in the identification image, and to delineate an identification frame for each coverage area according to the boundary line of each coverage area; it is also used to merge the identification frames of the first coverage areas to form a unique identification frame if there are several overlapping first coverage areas; it is also used to consider that the associated goods corresponding to the second coverage areas are stacked if there are several overlapping and non-overlapping second coverage areas, determine the upper and lower stacking order of the stacked associated goods, and adjust the display form of the identification frame corresponding to each second coverage area in sequence to determine the upper and lower stacking order corresponding to the second coverage area through the display form of the identification frame.

[0063] Optionally, it also includes a stacking stability determination module, which is used to determine and output the stacking stability based on the overlapping parts when there are several second coverage areas that overlap and do not overlap with each other; if the stacking stability does not meet the preset stability requirement, a preset mark is added to the identification box of the corresponding second coverage area, and the preset mark is used to indicate that there are goods under the associated goods corresponding to the corresponding identification box that are completely obscured by the associated goods and cannot be identified and displayed in the identification image.

[0064] Optionally, the identification frame demarcation module 1 is also used to select a minimum size from the sizes of all preset goods, and divide each identification area into a number of screening blocks based on the minimum size, and the size of each screening block is consistent with the minimum size; it is also used to select a balancing block in each screening block, and it satisfies: each screening block contains at least two or more balancing blocks.

[0065] An embodiment of the present application also discloses a warehouse cargo monitoring and identification device based on image processing technology. The warehouse cargo monitoring and identification device based on image processing technology includes a memory and a processor. The memory stores a computer program that can be loaded by the processor and executes the warehouse cargo monitoring and identification method based on image processing technology as described above.

[0066] An embodiment of the present application also discloses a computer-readable storage medium, which stores a computer program that can be loaded by a processor and executes the warehouse cargo monitoring and identification method based on image processing technology as described above. The computer-readable storage medium includes, for example: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0067] It should be noted that, in this document, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0068] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the protection scope of the application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on these embodiments, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

Claims

1. A warehouse cargo monitoring and identification method based on image processing technology, characterized in that: include: Obtaining a monitoring instruction, capturing a recognition image, delineating a light-dark dividing line from the recognition image, performing a light-dark balancing process on an area in the recognition image that is divided by each light-dark dividing line, and delineating a recognition frame in the recognition image after the light-dark balancing process; The area image in the recognition frame in the recognition image is used as a recognition object, the recognition object is input into a pre-built recognition model, and a recognition result is output through the recognition model; wherein the recognition result at least includes the position information of the recognition frame, the determination result of whether there are goods in the warehouse area corresponding to the recognition frame, and the type of goods when there are goods in the area corresponding to the recognition frame; Training images and verification images are regularly collected, and the recognition model is optimized using the training images and verification images.

2. The warehouse cargo monitoring and identification method based on image processing technology according to claim 1 is characterized in that: The step of performing light-dark balancing processing on the area divided by each light-dark dividing line in the recognition image and defining a recognition frame in the recognition image after the light-dark balancing processing comprises: The area formed by the light-dark dividing line in the recognition image is used as the recognition area, and a balanced block is selected from each of the recognition areas; Using the pre-built brightness balance model, perform brightness balance on the image fragments corresponding to each balanced block to obtain a deduced image, compare the deduced image with the reference image in the pre-stored image library, and if there is a reference image containing the deduced image, then the balance is completed, and the deduced image is used to replace the image corresponding to the corresponding balanced block in the identified image, and the goods corresponding to the reference image are used as the associated goods of the balanced block; Based on the associated goods, all balanced blocks are fused and delineated to form an identification frame.

3. The warehouse cargo monitoring and identification method based on image processing technology according to claim 2 is characterized in that: The method of using the pre-built brightness equalization model to perform brightness equalization on the image fragments corresponding to each equalization block to obtain a deduced image, and comparing the deduced image with a reference image in a pre-stored image library. If there is a reference image containing the deduced image, completing the equalization includes: Determine a first equalization value according to the position information of each equalization block in the identified image, and for each equalization block, perform shallow equalization processing on the image fragments corresponding to the equalization block through a pre-constructed brightness equalization model according to the corresponding first equalization parameter to obtain a deduced image; According to the position information of each equalization block in the identified image, a second equalization value corresponding to the equalization block is determined; for each equalization block, according to the corresponding second equalization parameter, a deep equalization process is performed on the corresponding deduced image through the brightness equalization model, and the deduced image after each deep equalization process is compared with a reference image in a pre-stored image library; If there is a reference image that includes a deduced image after deep equalization processing, the deep equalization processing of the corresponding equalization block is completed; if not, the second equalization parameter corresponding to the equalization block is re-determined, and the corresponding deduced image is deep-equalized again through the brightness equalization model until there is a reference image that includes a deduced image after deep equalization processing.

4. The warehouse cargo monitoring and identification method based on image processing technology according to claim 3 is characterized in that: The determining of the second equalization parameter corresponding to the equalization block includes: According to the pre-stored cargo attributes and the position information of the balancing block in the recognition image, the shadow projection degrees of all preset cargoes in the recognition area where the balancing block is located are determined, a second balancing data set is determined for the balancing block, and a deep balancing algorithm is generated for the balancing block based on the second balancing data set; wherein the cargo attributes at least include the light transmittance of the cargo outer packaging material, and the second balancing data set refers to a data set consisting of a number of shadow projection degrees; The deep equalization processing of the corresponding deduced image by using the brightness equalization model includes: The deep equalization algorithm is loaded through the brightness equalization model to perform deep equalization processing on the corresponding deduced image.

5. The warehouse cargo monitoring and identification method based on image processing technology according to claim 2 is characterized in that: Based on the associated goods, all the balanced blocks are merged and delineated to form an identification frame, including: Determine the coverage area of ​​the associated goods corresponding to each balancing block in the identification image according to the size of the associated goods corresponding to each balancing block and the position information of each balancing block in the identification image, and define an identification frame for each coverage area according to the boundary line of each coverage area; If there are several mutually overlapping first coverage areas, the identification frames of the first coverage areas are merged to form a unique identification frame; If there are several second coverage areas that overlap and do not overlap with each other, it is considered that the associated goods corresponding to the second coverage areas are stacked on each other, and the upper and lower stacking order of the stacked associated goods is determined, and the display form of the identification box corresponding to each second coverage area is adjusted in sequence to determine the upper and lower stacking order corresponding to the second coverage area through the display form of the identification box.

6. The warehouse cargo monitoring and identification method based on image processing technology according to claim 5 is characterized in that: The method further comprises: When there are a plurality of second coverage areas that overlap each other and do not overlap each other, determining and outputting stacking stability based on the overlapping parts; If the stacking stability does not meet the preset stability requirement, a preset mark is added at the identification frame of the corresponding second coverage area, and the preset mark is used to indicate that there are goods under the associated goods corresponding to the corresponding identification frame that are completely blocked by the associated goods and cannot be identified and displayed in the identification image.

7. The warehouse cargo monitoring and identification method based on image processing technology according to claim 2 is characterized in that: The step of selecting a balanced block from each of the identified areas comprises: Selecting a minimum size from all preset cargo sizes, and dividing each identification area into a number of screening blocks according to the minimum size, wherein the size of each screening block is consistent with the minimum size; A balancing block is selected in each screening block, and the following conditions are satisfied: each screening block contains at least two or more balancing blocks.

8. A warehouse cargo monitoring and identification system based on image processing technology, characterized in that: include, The recognition frame delineation module (1) is used to obtain monitoring instructions, capture a recognition image, delineate light and dark dividing lines from the recognition image, perform light and dark balancing processing on the area in the recognition image divided by each light and dark dividing line, and delineate a recognition frame in the recognition image after the light and dark balancing processing; A cargo identification module (2) is used to use the area image in the identification image and within the identification frame as an identification object, input the identification object into a pre-built identification model, and output a recognition result through the identification model; wherein the recognition result at least includes the position information of the identification frame, the determination result of whether there is cargo in the warehouse area corresponding to the identification frame, and the type of cargo when there is cargo in the area corresponding to the identification frame; The model optimization module (3) is used to regularly collect training images and verification images, and optimize the recognition model using the training images and verification images.

9. A warehouse cargo monitoring and identification device based on image processing technology, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute any one of the methods according to claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute any one of the methods as claimed in claims 1 to 7.

Images