Intelligent monitoring method and system for beverage production line based on digital analysis

Abstract

The application relates to the field of production line monitoring, in particular to a beverage production line intelligent monitoring method and system based on digital analysis, which performs image acquisition on a product to be detected on a production line, acquires N product top views, and obtains a product segmentation image corresponding to each product top view; the N product segmentation images are superimposed to obtain one superimposed image; according to a first target area image and a second target area image divided in the superimposed image, a reference product area image and an error area image are determined; according to pixel points in the reference product area image, the error area image and the first target area image, abnormal monitoring and early warning are performed on the product to be detected; abnormal scoring is performed according to the reference product area and the error area; abnormal evaluation is performed from two directions of horizontal and vertical dimensions; compared with a conventional mode, abnormal conditions influencing the operation of the production line can be more accurately analyzed, so that the accuracy of intelligent monitoring of the production line is improved.

Description

Technical Field

[0001] This invention relates to the field of production line monitoring, and in particular to an intelligent monitoring method and system for beverage production lines based on digital analysis. Background Technology

[0002] With the rapid development of artificial intelligence technology, production line monitoring without human intervention has been gradually applied to manufacturing scenarios. Existing technologies typically extract feature point information of the product to be inspected in the production line, and then compare the feature point information of the product to be inspected with the feature point information of the reference product to determine whether the product to be inspected has abnormal product pose, thereby realizing intelligent monitoring of the production line.

[0003] In manufacturing scenarios, there are various environmental noise influences, such as conveyor belt vibration and placement position deviation. In such cases, they do not actually affect the normal operation of the production line. However, in cases such as product marking, when the product's posture deviates (i.e., the product is tilted due to the instability of the conveyor belt surface), it will actually affect the operation of the production line. However, in this case, it is very easy to make misjudgments of abnormal product posture by comparing feature point information. Therefore, how to improve the accuracy of intelligent monitoring of the production line has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and system for intelligent monitoring of beverage production lines based on digital analysis, in order to solve the problem of how to improve the accuracy of intelligent monitoring of production lines.

[0005] In a first aspect, embodiments of the present invention provide a production line intelligent monitoring method based on digital analysis, the production line intelligent monitoring method comprising:

[0006] Based on a preset sampling frequency, images of the products to be inspected on the production line are acquired, resulting in N product top views, where N≥2. Semantic segmentation is then performed on the N product top views to obtain product segmentation images corresponding to each product top view.

[0007] Following the acquisition order of the N product top views, the N product segmentation images are superimposed to obtain a single superimposed image;

[0008] The superimposed image is binarized using a preset first threshold. After binarization, a first target region image is defined in the superimposed image. The image outside the first target region image in the superimposed image is binarized using a preset second threshold. A second target region image is defined accordingly. Based on the first target region image and the second target region image, a reference product region image and an error region image in the superimposed image are determined.

[0009] Based on the number of pixels in the reference product area image and the error area image, a horizontal offset anomaly score value for the product to be detected is calculated. Based on all pixel values ​​in the first target area image, a vertical offset anomaly score value for the product to be detected is calculated. Anomaly monitoring and early warning are performed on the product to be detected based on the horizontal offset anomaly score value and the vertical offset anomaly score value.

[0010] Secondly, embodiments of the present invention provide a production line intelligent monitoring system based on digital analysis, the production line intelligent monitoring system comprising:

[0011] The image preprocessing module is used to acquire images of the products to be inspected on the production line based on a preset sampling frequency, and acquire N product top views, where N≥2. The module performs semantic segmentation on the N product top views to obtain the product segmentation image corresponding to each product top view.

[0012] The image overlay module is used to overlay the N product segmentation images according to the acquisition order of the N product top views to obtain an overlay image;

[0013] The region segmentation module is used to perform binarization processing on the superimposed image using a preset first threshold, and then delineate a first target region image in the superimposed image after processing. It also uses a preset second threshold to perform binarization processing on the image outside the first target region image in the superimposed image, and correspondingly delineate a second target region image. Based on the first target region image and the second target region image, it determines a reference product region image and an error region image in the superimposed image.

[0014] The anomaly warning module is used to calculate the horizontal offset anomaly score of the product to be detected based on the number of pixels in the reference product area image and the error area image, calculate the vertical offset anomaly score of the product to be detected based on all pixel values ​​in the first target area image, and perform anomaly monitoring and warning for the product to be detected based on the horizontal offset anomaly score and the vertical offset anomaly score.

[0015] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0016] This invention acquires images of products to be inspected on a production line based on a preset sampling frequency, obtaining N product top views (N≥2). Semantic segmentation is performed on the N product top views to obtain a segmented product image corresponding to each top view. The N product segmented images are then superimposed according to the acquisition order of the N product top views to obtain a superimposed image. A preset first threshold is used to binarize the superimposed image, and a first target region image is defined within the superimposed image. A preset second threshold is used to binarize the images outside the first target region image in the superimposed image, correspondingly defining a second target region image. Based on the first and second target region images, a reference product region image and an error region image are determined in the superimposed image. A horizontal offset anomaly score is calculated based on the number of pixels in the reference and error region images. A vertical offset anomaly score is calculated based on all pixel values ​​in the first target region image. Anomaly monitoring and early warning are then performed on the product to be inspected based on the horizontal and vertical offset anomaly scores. This method involves continuously acquiring multiple top-view images of the product, and then using forgetting overlay and thresholding to determine the reference product area and error area in the overlaid image. This allows the thresholding process to adapt to situations where there are large differences in pixel values ​​in the overlaid image, and it can also adaptively determine the accurate reference product area based on the actual production line operation. Therefore, it eliminates the need to pre-set the pose of the template product and has high robustness to noise during the placement process and the conveyor belt transmission process. Furthermore, it performs anomaly scoring based on the reference product area and error area, and evaluates anomalies from both horizontal and vertical dimensions. Compared with conventional methods, it can more accurately analyze anomalies affecting the production line operation, thereby improving the accuracy of intelligent monitoring of the production line. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of an application environment for an intelligent production line monitoring method based on digital analysis provided in Embodiment 1 of the present invention;

[0019] Figure 2 This is a flowchart illustrating a production line intelligent monitoring method based on digital analysis provided in Embodiment 1 of the present invention.

[0020] Figure 3This is a schematic diagram of the structure of a production line intelligent monitoring method system based on digital analysis provided in Embodiment 2 of the present invention;

[0021] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0022] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0023] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0024] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0026] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0028] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0029] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0030] The present invention provides a method for intelligent monitoring of production lines based on digital analysis, which can be applied to applications such as... Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0031] See Figure 2 This is a flowchart illustrating a digital analysis-based intelligent monitoring method for beverage production lines provided in Embodiment 1 of the present invention. The aforementioned intelligent monitoring method for production lines can be applied to... Figure 1 The client and server-side computer devices connect to the corresponding database to obtain multiple top-view images of the products to be inspected on the production line. These images are used to monitor pose deviation and provide early warning.

[0032] like Figure 2 As shown, the intelligent monitoring method for the production line may include the following steps:

[0033] Step S201: Based on a preset sampling frequency, images of the products to be inspected on the production line are acquired, resulting in N product top views (N≥2). Semantic segmentation is performed on the N product top views to obtain the product segmentation image corresponding to each product top view.

[0034] Here, sampling frequency defines the number of samples extracted from a continuous signal and used to assemble a discrete signal per unit time. The product to be inspected refers to a product for which pose anomaly detection has not been performed. A product top-view is an image taken from above the product. Semantic segmentation is used to divide an image into regions belonging to different semantic categories. A product segmentation image is the image corresponding to the segmented product from the product top-view.

[0035] In this embodiment of the invention, a camera is positioned above the conveyor belt, enabling it to capture top-down views of the products on the conveyor belt surface. Since the conveyor belt typically operates at a constant speed, the preset sampling frequency can be determined based on the conveyor belt's speed and the frequency at which products are placed on it. This ensures the camera can capture the corresponding top-down view of the product at each sampling moment. For example, a preset sampling frequency of 1 second is set, meaning one top-down view of a product is captured every second. Based on this preset sampling frequency, N top-down views of the product are captured within a time period T, where N ≥ 2. For any given top-down view of the product, semantic segmentation is performed to isolate the product, resulting in a segmented product image that eliminates background interference.

[0036] Optionally, the process involves acquiring images of the products to be inspected on the production line based on a preset sampling frequency, resulting in N top-view images of the products, including:

[0037] For any sampling time, acquire M top view images of the product to be tested at the sampling time, and stitch the M top view images to obtain the top view of the product at the sampling time, where M≥2;

[0038] Based on a preset sampling frequency, obtain the product top view at N sampling times, resulting in N product top views.

[0039] In this embodiment of the invention, cameras are arranged at various angles above the conveyor belt, enabling the cameras to capture complete images of the products on the conveyor belt surface, avoiding incomplete product images caused by camera blind spots. For any given sampling time, M cameras simultaneously capture images of the product to be inspected on the conveyor belt, obtaining M top-view images. These M top-view images are then stitched together to obtain the top-view of the product at that sampling time, where M ≥ 2. Similarly, according to a preset sampling frequency, N top-view images of the product are collected within a time period T.

[0040] Optionally, the step of semantically segmenting the N product top views to obtain a product segmentation image corresponding to each product top view includes:

[0041] Obtain the trained semantic segmentation model;

[0042] For any product top view, the trained semantic segmentation model is used to perform semantic segmentation on the product top view, resulting in a corresponding product segmentation image.

[0043] In this embodiment of the invention, the semantic segmentation model adopts an encoder-decoder structure. The input of the encoder is a top-view image of the product, and the output of the encoder is the semantic features of the top-view image of the product. The input of the decoder is the semantic features of the top-view image of the product, and the output of the decoder is a product segmentation image. In the product segmentation image, the pixel value of the pixel belonging to the product to be detected is 1, and the pixel value of other pixels is 0.

[0044] By training the semantic segmentation model, a trained semantic segmentation model is obtained. The specific training process is a conventional technique and will not be described in detail in this embodiment. Therefore, for any product top view, the trained semantic segmentation model is used to perform semantic segmentation on the product top view, resulting in a corresponding product segmentation image.

[0045] Step S202: Following the acquisition order of the N product top views, the N product segmentation images are superimposed to obtain a superimposed image.

[0046] In this context, an overlay image refers to an image obtained by superimposing one image onto another. Therefore, in this embodiment of the invention, based on the acquisition order of N product top views (i.e., the order from first to last acquisition), N product segmentation images are overlaid to obtain an overlay image. The overlapping areas, historical product pose areas, and current product pose areas of the multiple product segmentation images can be obtained based on the pixel value distribution in the overlay image. Through image overlay, it is not necessary to pre-select or set a standard product as a template; instead, a reference product can be adaptively determined for any product shape.

[0047] Optionally, the step of superimposing the N product segmentation images according to the acquisition order of the N product top views to obtain a single superimposed image includes:

[0048] According to the acquisition order of the N product top views, the first product segmentation image is used as the first target image, and the second product segmentation image is used as the second target image;

[0049] Align the first target image and the second target image pixel by pixel. For any pixel in the second target image, weighted superposition of the pixel and the target pixel is performed to obtain the output pixel corresponding to the position. The target pixel is the pixel in the first target image that is at the same position as the pixel.

[0050] A temporary overlay image between the first target image and the second target image is constructed based on the output pixels corresponding to all positions of the second target image;

[0051] The temporary overlay image is used as the first target image, and the third product segmentation image is used as the second target image. The steps of forming the temporary overlay image between the first target image and the second target image are repeated until all product segmentation images are overlaid to obtain a single overlay image.

[0052] In this embodiment of the invention, following the acquisition order of N product top views, the first product segmentation image is used as the first target image A, and the second product segmentation image is used as the second target image B. A weighted superposition calculation formula is used: C = αA + (1-α)B, to obtain a temporary superimposed image C, where α is the weight of the first target image and (1-α) is the weight of the second target image. Then, the temporary superimposed image is used as the first target image, and the third product segmentation image is used as the second target image. The weighted superposition formula is then used to repeatedly acquire temporary superimposed images, and so on, until all product segmentation images are superimposed to obtain a single superimposed image.

[0053] It is worth noting that since the product pose is less likely to be abnormal due to conveyor belt vibration and other reasons when the images are initially acquired, the weight of the corresponding product segmentation image is smaller during the image overlay process. Therefore, in this embodiment of the invention, α is preferably set to 0.05.

[0054] Step S203: The superimposed image is binarized using a preset first threshold. After processing, a first target region image is defined in the superimposed image. The image outside the first target region image in the superimposed image is binarized using a preset second threshold. A second target region image is defined accordingly. Based on the first target region image and the second target region image, the reference product region image and the error region image in the superimposed image are determined.

[0055] In this embodiment of the invention, the superimposed image is first thresholded using a preset first threshold to obtain a first binarized image. The region formed by pixels with a pixel value of 1 in the first binarized image is used as the first target region image. Then, the image outside the first target region image in the superimposed image is thresholded using a preset second threshold to obtain a second binarized image. The region formed by pixels with a pixel value of 1 in the second binarized image is used as the second target region image.

[0056] It is worth noting that the preset first threshold is mainly used to find overlapping areas, which are considered to be areas where all products have corresponding pixels when they are photographed. Therefore, theoretically, it is only necessary to find points with a pixel value of 1. However, in this embodiment, the products have a vertical offset, so the first threshold is appropriately reduced. The preferred preset first threshold is 0.8. The preset second threshold needs to be implemented through an adaptive threshold. Since this embodiment is a scenario without prior information, the second target area image can only be determined by the actual superposition result. Therefore, the preset second threshold can be determined using the Otsu thresholding method or other adaptive thresholding methods.

[0057] After determining the first target region image and the second target region image in the overlay image, the reference product region image and the error region image in the overlay image are determined based on the first target region image and the second target region image. In a staged determination manner, even if the thresholding process can adapt to the situation where the pixel values ​​in the overlay image are large, the accurate reference product region is adaptively determined according to the actual production line operation. Therefore, there is no need to pre-set the pose of the template product, and it has high robustness to the noise of the placement process and the noise of the conveyor belt transmission process.

[0058] Optionally, determining the reference product region image and the error region image in the overlaid image based on the first target region image and the second target region image includes:

[0059] The region image formed by the first target region image and the second target region image in the superimposed image is used as the reference product region image;

[0060] Obtain all non-zero pixels outside the reference product area image in the overlay image, and use the area image composed of all non-zero pixels as the error area image.

[0061] In this embodiment of the invention, since both the first target region image and the second target region image pertain to the position of the product to be detected on the conveyor belt, after determining the first and second target region images in the overlay image, the region image formed by the first and second target region images in the overlay image is used as the reference product region image, which is the position image corresponding to the product to be detected when there is no offset on the conveyor belt. Correspondingly, the region image formed by all non-zero pixels outside the reference product region image in the overlay image is used as the error region image, which is the region image corresponding to the horizontal position offset of the product.

[0062] Optionally, the step of obtaining the preset second threshold includes:

[0063] Obtain all non-zero pixels outside the first target region image in the overlay image, and calculate the average pixel value of all non-zero pixels as a preset second threshold.

[0064] In this embodiment of the invention, by acquiring all non-zero pixels outside the first target region image in the overlay image and calculating the average pixel value of all non-zero pixels as a preset second threshold, the second threshold for thresholding is determined based on the pixel values ​​of the pixels in the actual image, making the determined second threshold more appropriate.

[0065] Step S204: Calculate the horizontal offset anomaly score of the product to be tested based on the number of pixels in the reference product area image and the error area image; calculate the vertical offset anomaly score of the product to be tested based on all pixel values ​​in the first target area image; and perform anomaly monitoring and early warning for the product to be tested based on the horizontal offset anomaly score and the vertical offset anomaly score.

[0066] In this embodiment of the invention, a horizontal offset anomaly score value for the product to be detected is calculated based on the number of pixels in the reference product area image and the error area image. This score value is used to characterize the degree of offset of the product from the conveyor belt plane in the production line. A vertical offset anomaly score value for the product to be detected is calculated based on all pixel values ​​in the first target area image. This score value is used to characterize the degree of offset of the product from the conveyor belt plane in the production line. Both the horizontal and vertical offsets (such as curled edges) of the product to be detected are considered, making the subsequent anomaly detection results more accurate.

[0067] The mean between the horizontal offset anomaly score and the vertical offset anomaly score is calculated to obtain the anomaly score threshold. The anomaly score threshold is set to 0.7. The mean is compared with the anomaly score threshold. When the mean is greater than the anomaly score threshold, an anomaly monitoring warning for the product to be inspected is generated. Anomaly scoring is performed based on the reference product area and the error area. Anomaly evaluation is performed in both horizontal and vertical dimensions. Compared with conventional methods, it can more accurately analyze the abnormal situations affecting the production line operation, thereby improving the accuracy of intelligent monitoring of the production line.

[0068] Optionally, the step of calculating the horizontal offset anomaly score of the product to be detected based on the number of pixels in the reference product area image and the error area image includes:

[0069] The number of first pixels in the reference product area image and the number of second pixels in the error area image are obtained respectively. The sum of the number of first pixels and the number of second pixels is calculated. The sum is used as the denominator and the number of second pixels is used as the numerator to obtain the corresponding ratio. The ratio is used as the horizontal offset anomaly score of the product to be detected.

[0070] In this embodiment of the invention, the expression for calculating the horizontal offset anomaly score of the product to be tested is as follows:

[0071]

[0072] Where β1 is the horizontal offset anomaly score, S1 is the number of second pixels in the error region image, and S2 is the number of first pixels in the reference product region image.

[0073] It is worth noting that the more second pixels there are in the error area image, the larger the offset area of ​​the product parallel to the conveyor belt plane, and the greater the degree of offset. As a result, the higher the horizontal offset anomaly score, the more serious the horizontal offset of the product.

[0074] Optionally, the step of calculating the vertical offset anomaly score of the product to be detected based on all pixel values ​​in the first target region image includes:

[0075] Calculate the variance of pixel values ​​for all pixels in the first target region image, using a preset constant as the base and the negative of the pixel value variance as the exponent to obtain the corresponding exponential function result, and use the exponential function result as the vertical offset anomaly score of the product to be detected.

[0076] In this embodiment of the invention, the expression for calculating the vertical offset anomaly score of the product to be tested is as follows:

[0077]

[0078] Where β2 is the vertical offset anomaly score, e is a natural number, and σ 2 denoted as the variance of pixel values ​​for all pixels in the first target region image.

[0079] It is worth noting that the smaller the variance of pixel values ​​of all pixels in the first target area image, the smaller the overlapping area in the superimposed image, the more severe the vertical offset such as warping or curling of the product to be detected, and the larger the corresponding vertical offset anomaly score value.

[0080] Optionally, the step of monitoring and issuing early warnings for anomalies in the product under test based on the horizontal offset anomaly score and the vertical offset anomaly score includes:

[0081] Using preset horizontal deviation confidence weights and vertical deviation confidence weights, the horizontal deviation anomaly score and the vertical deviation anomaly score are weighted and summed to obtain the comprehensive anomaly score of the product to be tested, wherein the horizontal deviation confidence weight is greater than the vertical deviation confidence weight.

[0082] Anomalies are monitored and early warnings are issued for the product to be tested based on the comprehensive anomaly score.

[0083] In this embodiment of the invention, the calculation expression for the comprehensive anomaly score is as follows:

[0084]

[0085] Where γ is the overall anomaly score, f is the confidence weight of the horizontal deviation, and g is the confidence weight of the vertical deviation.

[0086] It is worth noting that since horizontal offset has a greater impact on the production line, for example, in product marking scenarios, horizontal offset can lead to incorrect marking positions, while vertical offset is more likely to cause scaling of the marking size. Therefore, the confidence weight of horizontal deviation is greater than that of vertical deviation.

[0087] When the overall anomaly score exceeds the anomaly score threshold, an anomaly monitoring and early warning for the product under test is generated, thus realizing the anomaly monitoring and early warning for the product under test.

[0088] This invention, in its embodiments, acquires images of products to be inspected on a production line based on a preset sampling frequency, obtaining N product top-view images (N≥2). Semantic segmentation is performed on the N product top-view images to obtain product segmentation images corresponding to each top-view image. The N product segmentation images are then superimposed according to the acquisition order of the N product top-view images to obtain a superimposed image. A preset first threshold is used to binarize the superimposed image, and a first target region image is defined within the superimposed image. A preset second threshold is used to binarize the images outside the first target region image in the superimposed image, correspondingly defining a second target region image. Based on the first and second target region images, a reference product region image and an error region image are determined in the superimposed image. The horizontal offset anomaly score of the product to be inspected is calculated based on the number of pixels in the reference and error region images. The vertical offset anomaly score of the product to be inspected is calculated based on all pixel values ​​in the first target region image. Anomaly monitoring and early warning are then performed on the product to be inspected based on the horizontal and vertical offset anomaly scores. This method involves continuously acquiring multiple top-view images of the product, and then using forgetting overlay and thresholding to determine the reference product area and error area in the overlaid image. This allows the thresholding process to adapt to situations where there are large differences in pixel values ​​in the overlaid image, and it can also adaptively determine the accurate reference product area based on the actual production line operation. Therefore, it eliminates the need to pre-set the pose of the template product and has high robustness to noise during the placement process and the conveyor belt transmission process. Furthermore, it performs anomaly scoring based on the reference product area and error area, and evaluates anomalies from both horizontal and vertical dimensions. Compared with conventional methods, it can more accurately analyze anomalies affecting the production line operation, thereby improving the accuracy of intelligent monitoring of the production line.

[0089] Corresponding to the intelligent production line monitoring method in the above embodiments, Figure 3 This diagram illustrates a structural block diagram of a production line intelligent monitoring system based on digital analysis, as provided in Embodiment 2 of the present invention. (See also...) Figure 3 The intelligent monitoring system for this production line includes:

[0090] The image preprocessing module 31 is used to acquire images of the products to be inspected on the production line based on a preset sampling frequency, acquire N product top views, N≥2, perform semantic segmentation on the N product top views, and obtain product segmentation images corresponding to each product top view.

[0091] The image overlay module 32 is used to overlay the N product segmentation images according to the acquisition order of the N product top views to obtain an overlay image;

[0092] The region segmentation module 33 is used to perform binarization processing on the superimposed image using a preset first threshold, and after processing, to delineate a first target region image in the superimposed image. It also uses a preset second threshold to perform binarization processing on the image outside the first target region image in the superimposed image, and correspondingly delineates a second target region image. Based on the first target region image and the second target region image, it determines a reference product region image and an error region image in the superimposed image.

[0093] The anomaly warning module 34 is used to calculate the horizontal offset anomaly score value of the product to be detected based on the number of pixels in the reference product area image and the error area image, calculate the vertical offset anomaly score value of the product to be detected based on all pixel values ​​in the first target area image, and perform anomaly monitoring and warning for the product to be detected based on the horizontal offset anomaly score value and the vertical offset anomaly score value.

[0094] Optionally, the image overlay module 32 includes:

[0095] The image selection unit is used to select the first product segmentation image as the first target image and the second product segmentation image as the second target image according to the acquisition order of the N product top views;

[0096] The weighted superposition unit is used to align the first target image and the second target image with pixels. For any pixel in the second target image, the pixel is weighted and superimposed with a target pixel to obtain the output pixel corresponding to the position. The target pixel is the pixel in the first target image that is at the same position as the pixel.

[0097] An image composition unit is configured to construct a temporary overlay image between the first target image and the second target image based on the output pixels corresponding to all positions of the second target image;

[0098] The image traversal unit is used to take the temporary overlay image as the first target image, take the third product segmentation image as the second target image, and repeat the steps of forming a temporary overlay image between the first target image and the second target image until all product segmentation images are overlaid to obtain an overlay image.

[0099] Optionally, the region division module 33 includes:

[0100] The first constitutive unit is used to take the region image composed of the first target region image and the second target region image in the superimposed image as the reference product region image.

[0101] The second constitutive unit is used to acquire all non-zero pixels outside the reference product area image in the superimposed image, and to take the area image composed of all non-zero pixels as the error area image.

[0102] Optionally, the anomaly warning module 34 includes:

[0103] The horizontal anomaly detection unit is used to acquire the number of first pixels in the reference product area image and the number of second pixels in the error area image, calculate the sum of the number of first pixels and the number of second pixels, use the sum as the denominator and the number of second pixels as the numerator to obtain the corresponding ratio, and use the ratio as the horizontal offset anomaly score of the product to be detected.

[0104] Optionally, the anomaly warning module 34 includes:

[0105] The vertical anomaly detection unit is used to calculate the variance of pixel values ​​of all pixels in the first target region image, using a preset constant as the base and the negative of the pixel value variance as the exponent to obtain the corresponding exponential function result, and using the exponential function result as the vertical offset anomaly score value of the product to be detected.

[0106] Optionally, the anomaly warning module 34 includes:

[0107] The comprehensive anomaly detection unit is used to perform a weighted summation of the horizontal deviation anomaly score and the vertical deviation anomaly score using preset horizontal deviation confidence weights and vertical deviation confidence weights to obtain the comprehensive anomaly score of the product to be detected, wherein the horizontal deviation confidence weight is greater than the vertical deviation confidence weight.

[0108] The comprehensive early warning unit is used to monitor and issue early warnings for abnormalities in the product to be tested based on the comprehensive abnormality score.

[0109] Optionally, the image preprocessing module 31 includes:

[0110] The model acquisition unit is used to acquire the trained semantic segmentation model.

[0111] The model segmentation unit is used to perform semantic segmentation on any product top view using the trained semantic segmentation model, resulting in a corresponding product segmentation image.

[0112] Optionally, the image preprocessing module 31 includes:

[0113] The image stitching unit is used to acquire M top view images of the product to be detected at any sampling time, and to stitch the M top view images to obtain the top view of the product at the sampling time, where M≥2;

[0114] The image acquisition unit is used to acquire N product top views at N sampling times based on a preset sampling frequency, thus obtaining N product top views.

[0115] Optionally, the region division module 33 includes:

[0116] The threshold acquisition unit is used to acquire all non-zero pixels outside the first target region image in the superimposed image, and calculate the average pixel value of all non-zero pixels as a preset second threshold.

[0117] It should be noted that the information interaction and execution process between the above modules and units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0118] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the diagram), a memory, and a computer program stored in the memory and capable of running on at least one processor, which, when executing the computer program, implements the steps in any of the above embodiments of the intelligent production line monitoring method.

[0119] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0120] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0121] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0122] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0123] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0124] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0125] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0126] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0127] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A digital analysis-based production line intelligent monitoring method, characterized by, The production line intelligent monitoring method comprises: Based on a preset sampling frequency, image acquisition is performed on a product to be detected on a production line, N product top views are acquired, N 2. Semantic segmentation is performed on the N product top views, and a product segmentation image corresponding to each product top view is acquired. According to the collection sequence of the N product top views, superimpose the N product segmentation images to obtain a superimposed image; Using a preset first threshold value to perform binaryzation processing on the superimposed image, and after processing, a first target region image is demarcated in the superimposed image, using a preset second threshold value to perform binaryzation processing on the image outside the first target region image in the superimposed image, and a second target region image is demarcated, and according to the first target region image and the second target region image, a reference product region image and an error region image in the superimposed image are determined; According to the number of pixel points in the reference product region image and the error region image, a horizontal offset abnormal score value of the product to be detected is calculated, according to all pixel values in the first target region image, a vertical offset abnormal score value of the product to be detected is calculated, and the product to be detected is monitored and warned according to the horizontal offset abnormal score value and the vertical offset abnormal score value; According to the first target region image and the second target region image, the reference product region image and the error region image in the superimposed image are determined, comprising: The region image composed of the first target region image and the second target region image in the superimposed image is taken as the reference product region image; All non-zero pixel points outside the reference product region image in the superimposed image are obtained, and the region image composed of all non-zero pixel points is taken as the error region image; According to the number of pixel points in the reference product region image and the error region image, a horizontal offset abnormal score value of the product to be detected is calculated, comprising: The number of first pixel points in the reference product region image and the number of second pixel points in the error region image are obtained respectively, the addition result between the number of first pixel points and the number of second pixel points is calculated, the corresponding ratio is obtained by taking the addition result as the denominator and the number of second pixel points as the numerator, and the ratio is taken as the horizontal offset abnormal score value of the product to be detected; According to all pixel values in the first target region image, a vertical offset abnormal score value of the product to be detected is calculated, comprising: The pixel value variance of all pixel points in the first target region image is calculated, the corresponding exponential function result is obtained by taking a preset constant as the base and the inverse of the pixel value variance as the index, and the exponential function result is taken as the vertical offset abnormal score value of the product to be detected.

2. The production line intelligent monitoring method according to claim 1, characterized in that, According to the collection sequence of the N product top views, superimpose the N product segmentation images to obtain a superimposed image, comprising: According to the collection sequence of the N product top views, superimpose the N product segmentation images to obtain a superimposed image, comprising: According to the collection sequence of the N product top views, superimpose the N product segmentation images to obtain a superimposed image, comprising: The first target image and the second target image are pixel point aligned, for a pixel point at any position in the second target image, the pixel point and a target pixel point are weighted and superimposed to obtain an output pixel point corresponding to the position, the target pixel point is a pixel point in the first target image with the same position as the pixel point; An interim superimposed image between the first target image and the second target image is formed according to the output pixel points corresponding to all positions of the second target image; The interim superimposed image is taken as the first target image, a third product segmentation image is taken as the second target image, and the step of forming the interim superimposed image between the first target image and the second target image is repeated until all product segmentation images are superimposed, to obtain a superimposed image.

3. The production line intelligent monitoring method according to claim 1, wherein, The abnormal monitoring and early warning of the product to be detected according to the horizontal offset abnormal score value and the vertical offset abnormal score value comprises: The horizontal offset abnormal score value and the vertical offset abnormal score value are weighted and summed by using preset horizontal deviation confidence weight and vertical deviation confidence weight, to obtain a comprehensive abnormal score value of the product to be detected, wherein the horizontal deviation confidence weight is greater than the vertical deviation confidence weight; The product to be detected is monitored and warned abnormally according to the comprehensive abnormal score value.

4. The production line intelligent monitoring method according to claim 1, characterized in that, The semantic segmentation of the N product top views is performed to obtain product segmentation images corresponding to each product top view, comprising: A trained semantic segmentation model is obtained; For any product top view, the trained semantic segmentation model is used to perform semantic segmentation on the product top view, and a corresponding product segmentation image is obtained.

5. The production line intelligent monitoring method according to claim 1, wherein, The image acquisition of the product to be detected on the production line based on a preset sampling frequency is performed to obtain N product top views, comprising: For any sampling time, M top-view images of the product to be detected collected at the sampling time are acquired, the M top-view images are stitched to obtain a top-view image of the product at the sampling time, M 2; Based on a preset sampling frequency, product top views at N sampling time points are obtained, and N product top views are correspondingly obtained.

6. The production line intelligent monitoring method according to claim 1, wherein, The preset second threshold value is obtained by: All non-zero pixel points in the superimposed image except the first target area image are obtained, and the mean value of the pixel values of all non-zero pixel points is calculated as the preset second threshold value.

7. A digital analysis-based production line intelligent monitoring system, characterized by, The production line intelligent monitoring system comprises: An image preprocessing module is configured to collect images of products on a production line based on a preset sampling frequency, and N overhead views of the products are collected.

2. Perform semantic segmentation on the N overhead views of the products to obtain product segmentation images corresponding to each of the overhead views of the products, respectively. An image superimposition module is configured to superimpose the N product segmentation images in the order of the N product top views to obtain a superimposed image; A region division module is configured to perform binaryzation processing on the superimposed image by using a preset first threshold value, to define a first target area image in the superimposed image, perform binaryzation processing on the image in the superimposed image except the first target area image by using a preset second threshold value, to correspondingly define a second target area image, and determine a reference product area image and an error area image in the superimposed image according to the first target area image and the second target area image; An abnormality early warning module is configured to calculate a horizontal offset abnormality score of the product to be detected according to the number of pixel points in the reference product area image and the error area image, calculate a vertical offset abnormality score of the product to be detected according to all pixel values in the first target area image, and perform abnormality monitoring and early warning on the product to be detected according to the horizontal offset abnormality score and the vertical offset abnormality score. The area division module comprises: A first constituent unit is configured to take a region image formed by a first target area image and a second target area image in the superimposed image as a reference product area image; A second constituent unit is configured to obtain all non-zero pixel points in the superimposed image except the reference product area image, and take a region image formed by all the non-zero pixel points as an error area image; The abnormality early warning module comprises: A horizontal abnormality detection unit is configured to obtain a first number of pixel points in the reference product area image and a second number of pixel points in the error area image respectively, calculate an addition result between the first number of pixel points and the second number of pixel points, take the second number of pixel points as a numerator and the addition result as a denominator to obtain a corresponding ratio, and take the ratio as the horizontal offset abnormality score of the product to be detected; A vertical abnormality detection unit is configured to calculate a pixel value variance of all pixel points in the first target area image, take a preset constant as a base number, take the inverse of the pixel value variance as an index to obtain a corresponding exponential function result, and take the exponential function result as the vertical offset abnormality score of the product to be detected.

Images