Glass detection processing method and device and terminal equipment

Abstract

The invention is suitable for the technical field of machine vision, and provides a glass detection processing method and device and terminal device.The method comprises the steps that in response to a detection instruction of a user on to-be-detected glass, original image data of the to-be-detected glass are obtained, the original image data comprise red-green-blue image data and / or ultraviolet visible light image data, and the red-green-blue image data and / or the ultraviolet visible light image data are transmitted to the user; the glass to be detected is photovoltaic glass in a molten state; performing feature extraction on the original image data to obtain target feature data of the to-be-detected glass; and analyzing the target feature data to obtain a detection result of the to-be-detected glass, the detection result being used for representing whether foreign matters exist in the to-be-detected glass. According to the method, feature extraction is performed on the red, green and blue image data and / or the ultraviolet visible light image data of the to-be-detected glass, so that foreign matter detection of the glass liquid can be realized by penetrating through the surface layer of the glass liquid, missing detection caused by visual fatigue is avoided, the glass detection efficiency and accuracy are improved, and the product quality is improved.

Description

Technical Field

[0001] The present application belongs to the field of machine vision technology, and in particular relates to a processing method, device and terminal equipment for glass detection. Background Art

[0002] Photovoltaic glass is a material with high transparency. Its glass liquid remains clear and transparent at extremely high temperatures, which makes it impossible for relevant personnel to conduct direct close-up inspections.

[0003] The existing glass inspection method mainly involves adding monitoring equipment at the front end, and the inspection personnel observe through the display screen at the back end.

[0004] However, the above-mentioned glass inspection processing method mainly relies on human vision for observation and judgment, which is prone to missed detection due to visual fatigue. It is especially difficult to deal with complex defects, resulting in low inspection efficiency and a high missed detection rate, which seriously affects product quality. Summary of the Invention

[0005] The embodiments of the present application provide a glass detection processing method, apparatus, and terminal device, which can improve the efficiency and accuracy of glass detection while avoiding missed detection due to visual fatigue, thereby improving product quality.

[0006] In a first aspect, an embodiment of the present application provides a glass detection processing method, comprising:

[0007] In response to a user's inspection instruction for the glass to be inspected, obtaining raw image data of the glass to be inspected, wherein the raw image data includes red, green, and blue image data and / or ultraviolet and visible light image data, and the glass to be inspected is photovoltaic glass in a molten state;

[0008] Perform feature extraction on the original image data to obtain target feature data of the glass to be inspected;

[0009] The target feature data is analyzed to obtain the test results of the glass to be tested, and the test results are used to indicate whether there are foreign objects in the glass to be tested.

[0010] In a possible implementation of the first aspect, the detection instruction carries a target algorithm identifier, and feature extraction is performed on the original image data to obtain target feature data of the glass to be detected, including:

[0011] If the target algorithm identifier is detected as the first algorithm identifier, an improved YOLO target detection model is used to perform feature extraction on the original image data to obtain target feature data of the glass to be detected, wherein the image fusion processing module in the improved YOLO target detection model includes multiple transformer networks, and the image fusion processing module is used to perform fusion processing of the image data;

[0012] If the target algorithm identifier is detected to be the second algorithm identifier, feature extraction is performed on the original image data based on the historical image data of the glass to be detected to obtain target feature data of the glass to be detected, wherein the historical image data and the original image data are image data of the glass to be detected at different time nodes respectively.

[0013] In a possible implementation of the first aspect, the original image data includes red, green, and blue image data and ultraviolet and visible light image data. An improved YOLO target detection model is used to perform feature extraction on the original image data to obtain target feature data of the glass to be inspected, including:

[0014] Preprocessing the red, green, and blue image data and the ultraviolet and visible light image data respectively to obtain first processed image data and second processed image data, wherein the preprocessing at least includes a cropping process;

[0015] Using an improved YOLO target detection model to perform feature extraction on the first processed image data and the second processed image data to obtain extracted image data;

[0016] The extracted image data is subjected to format conversion processing to obtain target feature data of the glass to be detected. The format conversion processing is used to convert the format of the extracted image data into a preset format.

[0017] In a possible implementation of the first aspect, using an improved YOLO object detection model to perform feature extraction on the first processed image data and the second processed image data to obtain extracted image data includes:

[0018] Perform convolution processing on the first processed image data and the second processed image data respectively to obtain third processed image data and fourth processed image data;

[0019] inputting the third processed image data and the fourth processed image data into an image fusion processing module to obtain first fused image data and second fused image data, wherein the first processed image data and the first fused image data have the same spectral value, and the second processed image data and the second fused image data have the same spectral value;

[0020] performing an exclusive OR calculation on the third processed image data and the first fused image data to obtain fifth processed image data, and performing an exclusive OR calculation on the fourth processed image data and the second fused image data to obtain sixth processed image data;

[0021] Extracted image data is obtained by calculation based on the fifth processed image data and the sixth processed image data. The extracted image data at least includes a grayscale value.

[0022] In a possible implementation of the first aspect, inputting the third processed image data and the fourth processed image data into an image fusion processing module to obtain first fused image data and second fused image data includes:

[0023] performing splicing processing on the third processed image data and the fourth processed image data to obtain spliced ​​image data;

[0024] Performing an XOR calculation on the spliced ​​image data and the preset position code to obtain first calculation data;

[0025] Inputting the first calculated data into multiple transformer networks in the image fusion processing module in sequence to obtain target fused image data;

[0026] The target fused image data is split to obtain first fused image data and second fused image data.

[0027] In a possible implementation of the first aspect, each transformer network includes a first normalization layer, a multi-head attention mechanism unit, a second normalization layer, and a multilayer perceptron. The first calculation data is sequentially input into multiple transformer networks in the image fusion processing module to obtain target fused image data, including:

[0028] Normalizing the first calculated data based on the first normalization layer to obtain second calculated data;

[0029] Using a multi-head attention mechanism unit to perform attention mechanism processing on the second calculation data to obtain third calculation data;

[0030] Performing an XOR calculation on the first calculation data and the third calculation data to obtain fourth calculation data;

[0031] Normalizing the fourth calculated data based on the second normalization layer to obtain fifth calculated data;

[0032] Performing perception processing on the fifth calculation data using a multilayer perceptron to obtain sixth calculation data;

[0033] Performing an XOR operation on the fifth calculated data and the sixth calculated data to obtain seventh calculated data;

[0034] The seventh calculated data is used as the first calculated data, and the step of normalizing the first calculated data based on the first normalization layer is returned to obtain the second calculated data, until the first calculated data is input into all transformer networks to obtain the target fused image data.

[0035] In a possible implementation of the first aspect, the original image data includes red, green, and blue image data or ultraviolet and visible light image data, and the target feature data includes the number of target detection areas, and the movement distance and number of successful pixel movements of target pixels within the target detection areas;

[0036] Based on the historical image data of the glass to be inspected, feature extraction is performed on the original image data to obtain the target feature data of the glass to be inspected, including:

[0037] Calculate the grayscale difference between the original pixel points at the same position in the original image data and the historical image data to obtain a difference image;

[0038] Merge the original pixels in the difference image according to the grayscale difference of the original pixels in the difference image to generate a candidate detection area of ​​the original image data;

[0039] Position matching is performed on the candidate detection areas of the original image data and the candidate detection areas of the historical image data, and the candidate detection areas with intersecting positions are determined as target detection areas;

[0040] Determine the number of target detection areas, as well as the pixel coordinates and movement distances of target pixels within the target detection areas;

[0041] For each target pixel point in the target detection area, determine whether each target pixel point conforms to the foreign object movement law based on the pixel coordinates of each target pixel point;

[0042] The number of target pixels that conform to the foreign object movement law is determined as the number of successful pixel movements.

[0043] In a possible implementation of the first aspect, after analyzing the target feature data to obtain a test result of the glass to be tested, the method further includes:

[0044] If the detection result indicates that there is foreign matter in the glass to be detected, an alarm message of the glass to be detected is generated, and the alarm message is used to prompt the user to check the glass to be detected;

[0045] Marking foreign matter in the original image data based on the target feature data to obtain marked image data;

[0046] Send a video recording instruction to the image acquisition device, where the video recording instruction is used to instruct the image acquisition device to record the video of the glass to be inspected.

[0047] In a second aspect, an embodiment of the present application provides a glass detection processing device, comprising:

[0048] an acquisition module, configured to acquire raw image data of the glass to be inspected in response to a user's inspection instruction for the glass to be inspected, wherein the raw image data includes red, green, and blue image data and / or ultraviolet and visible light image data, and the glass to be inspected is photovoltaic glass in a molten state;

[0049] A feature extraction module is used to extract features from the original image data to obtain target feature data of the glass to be inspected;

[0050] The analysis module is used to analyze the target feature data to obtain the detection result of the glass to be detected. The detection result is used to indicate whether there is foreign matter in the glass to be detected.

[0051] In a third aspect, an embodiment of the present application provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods of the first aspect when executing the computer program.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of any one of the first aspects.

[0053] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the methods in the first aspect above.

[0054] The embodiments of the present application provide a processing method, apparatus and terminal device for glass detection, the method comprising: in response to a user's detection instruction for the glass to be detected, obtaining original image data of the glass to be detected, wherein the original image data includes red, green and blue image data and / or ultraviolet visible light image data, and the glass to be detected is photovoltaic glass in a molten state; performing feature extraction on the original image data to obtain target feature data of the glass to be detected; analyzing the target feature data to obtain a detection result of the glass to be detected, and the detection result is used to characterize whether there is foreign matter in the glass to be detected. Utilizing the above technical solution, by performing feature extraction on the red, green and blue image data and / or ultraviolet visible light image data of the glass to be detected, the target feature data of the glass to be detected is obtained, and the foreign matter detection of the glass liquid can be realized through the surface layer of the glass liquid, while avoiding missed detection due to visual fatigue, the efficiency and accuracy of glass detection are improved, thereby improving product quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0056] Figure 1 This is a flow chart of a glass detection processing method provided in one embodiment of the present application;

[0057] Figure 2 is a flow chart of a glass detection processing method provided by another embodiment of the present application;

[0058] Figure 3 This is a schematic diagram of the overall process of feature extraction provided by an embodiment of the present application;

[0059] Figure 4 This is a schematic diagram of the overall process of an image fusion processing module provided by an embodiment of the present application;

[0060] Figure 5 This is a flow chart of a detection algorithm based on a non-network model provided in one embodiment of the present application;

[0061] Figure 6 This is a flowchart of a multi-threaded process provided by an embodiment of the present application;

[0062] Figure 7 This is a structural block diagram of a glass detection processing device provided in one embodiment of the present application;

[0063] Figure 8 This is a structural diagram of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0064] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0065] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0066] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0067] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0068] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0069] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0070] The glass detection processing method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific type of terminal device.

[0071] It can be said that the current glass inspection method mainly relies on human vision for observation and judgment, which is prone to missed detection due to visual fatigue. It is especially difficult to deal with complex defects, resulting in low inspection efficiency and a high missed detection rate. It is difficult to adapt to high-speed production lines, and ultimately leads to defects that cannot be discovered and handled in a timely manner.

[0072] At the same time, the current glass inspection processing method can only observe larger surface defects and has difficulty capturing inclusions inside the molten glass; the algorithms involved are highly complex and lack real-time performance; they are sensitive to lighting conditions and are easily affected by the fluidity of the glass liquid and surface ripples.

[0073] Based on this, the embodiment of the present application provides a glass detection processing method. By introducing a visual detection algorithm, it realizes automatic identification and alarm of foreign matter generated by photovoltaic glass liquid, which not only improves the accuracy and efficiency of detection, but also greatly reduces the dependence on artificial vision, ensuring the continuous and stable operation of the production line.

[0074] Figure 1 This is a flow chart of a glass detection processing method provided by an embodiment of the present application. As an example and not a limitation, this method can be applied to terminal devices, such as Figure 1 As shown, the method includes:

[0075] S101 : Responding to a user's detection instruction for glass to be detected, obtaining original image data of the glass to be detected.

[0076] The original image data includes red, green, and blue image data and / or ultraviolet and visible light image data. The glass to be inspected is photovoltaic glass in a molten state, that is, the glass to be inspected can be understood as glass liquid in a molten state at high temperature.

[0077] In this embodiment, the terminal device can be designed with a simple and intuitive UI interactive interface, aiming to provide users with a convenient and efficient operating experience. For example, the entire UI interactive interface can adopt a simple and clear layout style, mainly divided into three functional areas: a display box area, an input operation area, and a mode selection and control area. In the UI interactive interface, the user can enter a URL, perform operations such as mode selection, and then click the "Start" button. The user triggers the detection instruction for the glass to be inspected, and the terminal device can respond to this detection instruction to obtain the original image data of the glass to be inspected. Subsequently, the defect detection step of the glass liquid in the video stream can be started according to the selected mode; if the user wants to stop the detection, the user can click the "Stop" button in the UI interactive interface.

[0078] S102: Extract features from the original image data to obtain target feature data of the glass to be inspected.

[0079] The target characteristic data can be used to characterize specific characteristics of the glass to be inspected, and different target characteristic data can characterize different characteristics of the glass to be inspected.

[0080] After obtaining the original image data of the glass to be inspected, feature extraction can be performed on the original image data to obtain target feature data of the glass to be inspected. The means and process of feature extraction in this embodiment are not limited. For example, feature extraction can be performed on the original image data according to an algorithm specified by the user, and the target feature data extracted by different algorithms may be different; or the terminal device can perform feature extraction on the original image data according to a preset strategy. The preset strategy can be a pre-configured feature extraction strategy, such as directly performing feature extraction on the original image data through a neural network model, or obtaining the target feature data of the glass to be inspected by performing certain calculations on the original image data. This embodiment does not limit this.

[0081] S103: Analyze the target feature data to obtain a detection result of the glass to be detected. The detection result is used to indicate whether there is foreign matter in the glass to be detected.

[0082] In this step, the detection results of the glass to be tested can be obtained by analyzing the target characteristic data. For example, it can be detected whether there are bubbles, impurities and other foreign matter in the glass to be tested. The specific analysis process can, for example, compare the target characteristic data with the preset standard to detect whether there are foreign matter in the glass to be tested. Different target characteristic data can correspond to different preset standards.

[0083] In addition, this embodiment can also correspond to different processing flows for different detection results. For example, when the detection result indicates that there is no foreign matter in the glass to be detected, it means that there is no defect in the glass to be detected. Then, it can return to the step of obtaining the original image data of the glass to be detected and continue to detect and process the next original image data of the glass to be detected.

[0084] In some embodiments, after analyzing the target feature data to obtain the inspection result of the glass to be inspected, the method further includes:

[0085] If the detection result indicates that there is foreign matter in the glass to be detected, an alarm message of the glass to be detected is generated, and the alarm message is used to prompt the user to check the glass to be detected;

[0086] Marking foreign matter in the original image data based on the target feature data to obtain marked image data;

[0087] Send a video recording instruction to the image acquisition device, where the video recording instruction is used to instruct the image acquisition device to record the video of the glass to be inspected.

[0088] In a specific embodiment, if the detection result indicates that there is a foreign object in the glass to be inspected, a corresponding alarm mechanism can be executed, such as triggering the generation of a voice alarm to remind relevant personnel to handle it; the target position of the detection can also be marked in the video screen to obtain the marked image data; video recording can also be turned on to save the video of the detection result, such as sending a video recording instruction to the image acquisition device to instruct the image acquisition device to record the video of the glass to be inspected.

[0089] This embodiment provides a glass inspection processing method. In response to a user's inspection instruction for the glass to be inspected, the method obtains raw image data of the glass to be inspected, wherein the raw image data includes red, green, and blue image data and / or ultraviolet-visible light image data, and the glass to be inspected is photovoltaic glass in a molten state; performs feature extraction on the raw image data to obtain target feature data of the glass to be inspected; and analyzes the target feature data to obtain an inspection result of the glass to be inspected, which is used to characterize whether there is foreign matter in the glass to be inspected. Utilizing this method, by extracting features from the red, green, and blue image data and / or ultraviolet-visible light image data of the glass to be inspected, the method can penetrate the surface layer of the glass liquid to detect foreign matter in the glass liquid. This method improves the efficiency and accuracy of glass inspection while avoiding missed detections due to visual fatigue, thereby improving product quality.

[0090] Figure 2 This is a flow chart of a glass detection processing method provided by another embodiment of the present application. In this embodiment, the detection instruction carries a target algorithm identifier, and performs feature extraction on the original image data to obtain the target feature data of the glass to be detected, which is further optimized as follows: if the target algorithm identifier is detected to be the first algorithm identifier, the improved YOLO target detection model is used to perform feature extraction on the original image data to obtain the target feature data of the glass to be detected, wherein the image fusion processing module in the improved YOLO target detection model includes multiple transformer networks, and the image fusion processing module is used to perform fusion processing on the image data; if the target algorithm identifier is detected to be the second algorithm identifier, the original image data is subjected to feature extraction based on the historical image data of the glass to be detected to obtain the target feature data of the glass to be detected, wherein the historical image data and the original image data are the image data of the glass to be detected at different time nodes, respectively. Figure 2 As shown, the method includes:

[0091] S201 : Responding to a user's detection instruction for the glass to be detected, obtaining original image data of the glass to be detected.

[0092] S202: If the target algorithm identifier is detected to be the first algorithm identifier, an improved YOLO target detection model is used to perform feature extraction on the original image data to obtain target feature data of the glass to be detected.

[0093] Among them, the image fusion processing module in the improved YOLO target detection model includes multiple transformer networks, and the image fusion processing module is used to perform fusion processing of image data.

[0094] In this embodiment, the detection instruction may carry a target algorithm identifier, which can be an identifier of a user-specified processing algorithm. For example, the target algorithm identifier can be a first algorithm identifier or a second algorithm identifier. The first algorithm identifier can be used to represent a network-based detection algorithm. A network-based detection algorithm can be understood as an algorithm that uses a deep neural network to extract features from image data through a feature extraction module. For example, if the target algorithm identifier is the first algorithm identifier, this embodiment may use an improved YOLO target detection model to extract features from the original image data to obtain target feature data for the glass to be inspected. This embodiment optimizes the network structure, specifically redesigning the YOLO feature extraction network to improve operational efficiency and performance by simplifying the network structure. The core module of the newly designed YOLO feature extraction network is the image fusion module, which is primarily composed of three transformer networks. This significantly improves the efficiency and accuracy of feature extraction while simplifying the network structure, effectively reducing computational complexity and runtime.

[0095] More specifically, the following is a specific embodiment of using the improved YOLO target detection model to extract features from original image data.

[0096] In some embodiments, the original image data includes red, green, and blue image data and ultraviolet and visible light image data. The improved YOLO target detection model is used to perform feature extraction on the original image data to obtain target feature data of the glass to be detected, including:

[0097] Preprocessing the red, green, and blue image data and the ultraviolet and visible light image data respectively to obtain first processed image data and second processed image data, wherein the preprocessing at least includes a cropping process;

[0098] Using an improved YOLO target detection model to perform feature extraction on the first processed image data and the second processed image data to obtain extracted image data;

[0099] The extracted image data is subjected to format conversion processing to obtain target feature data of the glass to be detected. The format conversion processing is used to convert the format of the extracted image data into a preset format.

[0100] In a specific implementation, the red, green, and blue image data and the ultraviolet and visible light image can first be used as pre-processed input data respectively. For the image data of these two different modalities, a cropping operation can be performed first. The purpose is to remove irrelevant edge parts in the image so that subsequent processing can focus on key areas. It also helps to reduce the amount of data processing and improve processing efficiency.

[0101] Furthermore, the improved YOLO target detection model can be used to extract features from the cropped RGB image and the ultraviolet-visible light image. The specific feature extraction process can be determined according to the structure of the improved YOLO target detection model.

[0102] In some embodiments, an improved YOLO target detection model is used to perform feature extraction on the first processed image data and the second processed image data to obtain extracted image data, including:

[0103] Perform convolution processing on the first processed image data and the second processed image data respectively to obtain third processed image data and fourth processed image data;

[0104] inputting the third processed image data and the fourth processed image data into an image fusion processing module to obtain first fused image data and second fused image data, wherein the first processed image data and the first fused image data have the same spectral value, and the second processed image data and the second fused image data have the same spectral value;

[0105] performing an exclusive OR calculation on the third processed image data and the first fused image data to obtain fifth processed image data, and performing an exclusive OR calculation on the fourth processed image data and the second fused image data to obtain sixth processed image data;

[0106] Extracted image data is obtained by calculation based on the fifth processed image data and the sixth processed image data. The extracted image data at least includes a grayscale value.

[0107] In a specific embodiment, the improved YOLO target detection model may include a convolution layer and an image fusion processing module. For example, the first processed image data and the second processed image data can be convolutionally processed according to the convolution layer to obtain third processed image data and fourth processed image data, and then the third processed image data and the fourth processed image data are fused by the image fusion processing module to obtain corresponding first fused image data and second fused image data, respectively. Finally, the first fused image data and the second fused image data can be further calculated according to the convolution layer to obtain extracted image data, wherein the extracted image data at least includes grayscale values.

[0108] Figure 3This is a schematic diagram of the overall process of feature extraction provided by an embodiment of the present application. Figure 3 As shown, first, the RGB image and the ultraviolet-visible light image can be cropped separately, and the cropped RGB image and the ultraviolet-visible light image enter their own independent convolution layer processing flow respectively to obtain third processed image data and fourth processed image data; then the third processed image data and the fourth processed image data are input into the image fusion module (i.e., the image fusion processing module) to obtain corresponding first fused image data and second fused image data, respectively. Subsequently, the third processed image data and the first fused image data can be XORed to obtain fifth processed image data, and the fourth processed image data and the second fused image data can be XORed to obtain sixth processed image data; the obtained fifth processed image data and sixth processed image data enter their own independent convolution layer processing flow again, and the image data obtained after the convolution calculation are XORed and enter the output layer; the output layer performs final processing and conversion on the processed feature map, so that it is output in a suitable form to form a final feature map (i.e., extracted image data), which contains rich feature information after the fusion of the RGB image and the ultraviolet-visible light image, which can be used for subsequent target detection, recognition and other tasks.

[0109] Figure 4 This is a schematic diagram of the overall process of an image fusion processing module provided by an embodiment of the present application. Figure 4 As shown, the specific processing flow of the image fusion processing module may include: first, the third processed image data (such as A in the figure) and the fourth processed image data (such as B in the figure) may be deformed and spliced ​​to obtain spliced ​​image data; then, the spliced ​​image data and the preset position code may be XORed to obtain first calculated data; then, the first calculated data may be sequentially input into multiple transformer networks in the image fusion processing module to obtain target fused image data; finally, the obtained target fused image data may be split and processed to obtain first fused image data corresponding to the first processed image data (such as A' in the figure), and second fused image data corresponding to the second processed image data (such as B' in the figure). The preset position code may be a pre-configured position code.

[0110] More specifically, each transformer network includes a first normalization layer, a multi-head attention mechanism unit, a second normalization layer and a multi-layer perceptron, and the first calculation data is sequentially input into multiple transformer networks in the image fusion processing module. The specific process of obtaining the target fused image data may include: normalizing the first calculation data based on the first normalization layer to obtain the second calculation data; using the multi-head attention mechanism unit to perform attention mechanism processing on the second calculation data to obtain the third calculation data; performing XOR calculation on the first calculation data and the third calculation data to obtain the fourth calculation data; normalizing the fourth calculation data based on the second normalization layer to obtain the fifth calculation data; performing perception processing on the fifth calculation data using a multi-layer perceptron to obtain the sixth calculation data; performing XOR calculation on the fifth calculation data and the sixth calculation data to obtain the seventh calculation data; using the seventh calculation data as the first calculation data, and returning to execute the step of normalizing the first calculation data based on the first normalization layer to obtain the second calculation data, until the first calculation data is input into all transformer networks to obtain the target fused image data.

[0111] S203: If the target algorithm identifier is detected to be the second algorithm identifier, feature extraction is performed on the original image data according to the historical image data of the glass to be detected to obtain target feature data of the glass to be detected.

[0112] The historical image data and the original image data are respectively the image data of the glass to be inspected at different time points.

[0113] The second algorithm identifier can be used to characterize the detection algorithm based on the non-network model. The detection algorithm based on the non-network model can include a specially designed difference calculation and filtering algorithm. For example, the detection algorithm can be understood as an algorithm that extracts features by subtracting the current frame image data from the reference frame image data, and then removes interference (such as video shaking due to heat waves) through a filtering algorithm to obtain target feature data.

[0114] Specifically, if the target algorithm identifier is detected as the second algorithm identifier, this embodiment can perform feature extraction on the original image data based on the historical image data of the glass to be inspected to obtain the target feature data of the glass to be inspected, wherein the historical image data and the original image data are image data of the glass to be inspected at different time points. On this basis, the difference between the current frame image data and the reference frame image data can be quickly calculated to accurately extract the feature data with the highest probability of representing defects within the glass liquid. At the same time, through the built-in filtering algorithm, interference factors such as video jitter caused by heat waves can be effectively removed, ensuring that the extracted feature data truly reflects the defects within the glass liquid, providing pure feature data for subsequent defect analysis.

[0115] In some embodiments, the original image data includes red, green, and blue image data or ultraviolet and visible light image data, and the target feature data includes the number of target detection areas, and the movement distance and number of successful pixel movements of target pixels within the target detection areas;

[0116] Based on the historical image data of the glass to be inspected, feature extraction is performed on the original image data to obtain the target feature data of the glass to be inspected, including:

[0117] Calculate the grayscale difference between the original pixel points at the same position in the original image data and the historical image data to obtain a difference image;

[0118] Merge the original pixels in the difference image according to the grayscale difference of the original pixels in the difference image to generate a candidate detection area of ​​the original image data;

[0119] Position matching is performed on the candidate detection areas of the original image data and the candidate detection areas of the historical image data, and the candidate detection areas with intersecting positions are determined as target detection areas;

[0120] Determine the number of target detection areas, as well as the pixel coordinates and movement distances of target pixels within the target detection areas;

[0121] For each target pixel point in the target detection area, determine whether each target pixel point conforms to the foreign object movement law based on the pixel coordinates of each target pixel point;

[0122] The number of target pixels that conform to the foreign object movement law is determined as the number of successful pixel movements.

[0123] Figure 5 This is a flow chart of a detection algorithm based on a non-network model provided by an embodiment of the present application, such as Figure 5 As shown, first, a frame of image data can be read, for example, by continuously reading frames from a video stream. The previous frame of the current frame is used as a reference frame. Parameters are initialized and a detection region is created, covering only the surface of the molten glass, with a mask used to eliminate interference. A difference can then be made between frames, such as by calculating the Y-channel pixel difference in brightness for the same pixel within the detection region between the current frame and the reference frame, to produce a difference image. This difference is particularly pronounced in areas with stones. Pixels with large differences can then be merged into blocks. Specifically, the difference can be calculated and graded, excluding blocks not in the detection area. Adjacent pixels with the same difference intensity can then be merged to generate candidate regions.

[0124] Secondly, search and matching can be performed, such as using linked list dynamic tracking technology for multi-frame matching, that is, matching the detection blocks corresponding to multiple frames to obtain a series of data on the change of detection targets over time, so as to enter the subsequent motion analysis. The specific search and matching process can include traversing the linked list (i.e., the candidate detection areas of the original image data and the candidate detection areas of the historical image data), comparing them based on the intersection of two adjacent frame blocks as the matching condition, and updating the node information if the match is successful, such as recording the size of the foreign object block (i.e., the number of areas), pixel position (i.e., pixel coordinates), etc. If the match fails, a new node is created, which can be considered a false alarm and the reading of the next frame can be continued.

[0125] Finally, motion analysis and judgment can be performed. For example, when the relevant data reaches a certain amount, it can be judged whether each pixel point conforms to the law of stone movement (that is, based on the pixel coordinates of each target pixel point, it can be judged whether each target pixel point conforms to the law of foreign body movement), such as judging whether the target is descending at a uniform speed. If it does not conform to the law of stone movement, the node can be processed, that is, it is considered to be a false alarm; if it conforms to the law of stone movement, the success count can be increased, and after a certain number of times, it can be preliminarily judged as a stone.

[0126] Correspondingly, when analyzing the target feature data, the number of matches, the number of successful movements and the descent distance can be comprehensively judged. If the preset standard requirements are met, it will be determined as a stone and an alarm will be issued, including voice broadcast, stone marking, video saving, etc.

[0127] S204: Analyze the target feature data to obtain a detection result of the glass to be detected. The detection result is used to indicate whether there is foreign matter in the glass to be detected.

[0128] This embodiment provides a glass detection processing method, which extracts features from original image data based on different algorithms adopted by users according to different production environments. This ensures that target feature data adapts to different production environments, provides an accurate data basis for subsequent analysis, and further improves the accuracy and efficiency of glass detection.

[0129] Figure 6 This is a flowchart of a multi-threaded process provided by an embodiment of the present application. Figure 6 As shown in the figure, the collaboration process of the main thread, decoding thread and encoding thread is demonstrated, thereby realizing the real-time detection, alarm and recording functions of the video stream, and improving the processing efficiency and response speed through hardware encoding and other methods. Among them, the main thread can take on the core detection algorithm execution and alarm mechanism triggering tasks, and is mainly responsible for coordinating the data interaction between various modules.

[0130] The main thread can first be used to input a URL, such as the user can input a URL in the main thread, which is a key step in obtaining the video source; the main thread can read a frame of image data, such as reading a frame of image data from the decoding queue for subsequent detection operations; the main thread can also detect the read image frame to determine whether there is a target (such as stones and other objects that need to be detected); the main thread can determine whether the target is detected, such as judging based on the detection results. If no target is detected, continue to loop and read the next frame for detection. If the target is detected, enter the alarm and recording process; the main thread can also alarm and record. For example, when the target is detected, the main thread can trigger an alarm and start the encoding thread for video recording.

[0131] The decoding thread can be used to capture video streams, which is the source of image data. The decoding thread can also be used to decode the captured video streams and convert the video data into processable image frames. The decoding thread can also be used to store the decoded image frames in the decoding queue, waiting for the main thread to read them. Each time the main thread reads the latest frame, the queue will be cleared to ensure that the latest image data is processed each time, ensuring the smoothness and real-time transmission of video frames.

[0132] When a defect is detected, this embodiment automatically starts the encoding thread. The encoding thread can be used to mark the target after receiving the start signal of the main thread so that the target position can be clearly displayed in the video; the encoding thread can also be used to encode the video containing the target using hardware encoding, using the high-efficiency performance of the hardware to reduce the encoding delay; the encoding thread can also be used to save the encoded video for subsequent viewing and analysis.

[0133] Furthermore, the glass inspection processing method of this embodiment involves multiple key components, each with unique functions and characteristics, providing strong support for the implementation of the visual inspection invention. The camera in the image acquisition module is a fundamental component. High-resolution cameras are responsible for capturing clear images and have high frame rates, meeting the image acquisition requirements of high-speed moving objects. Different camera types, such as industrial cameras and infrared cameras, can adapt to various environments and inspection targets, providing rich image information.

[0134] After receiving the image data, the edge computing server uses a multimodal data fusion algorithm to perform feature extraction and processing; then, the decision-making module of the edge computing server can analyze and judge based on the extracted features, compare them with preset standards, and obtain detection results, such as determining whether there are bubbles or impurities in the glass liquid.

[0135] The core of the edge computing server is a fusion algorithm chip with powerful computing capabilities that can quickly process data from multiple sensors. This chip can adopt advanced architecture design to optimize data processing, ensure the accurate fusion of different data, and improve information integrity and accuracy.

[0136] At the same time, deep neural network chips can be used in network model-based detection algorithms, where they play a crucial role. The chip integrates a large number of neurons and possesses high-speed computing power, enabling in-depth analysis of fused multimodal data to quickly and accurately extract key features of target objects. Furthermore, the chip supports flexible programming, allowing users to easily adjust the structure and parameters of the neural network based on different glass defect detection tasks, achieving efficient extraction of various complex defect features, thereby significantly improving detection accuracy and adaptability.

[0137] The edge computing server's processor is responsible for analyzing and judging the extracted features. It features an intelligent decision-making algorithm that quickly compares features with pre-set standards to make accurate judgments. The processor also supports a real-time feedback mechanism, outputting test results promptly to ensure the efficiency of the testing process.

[0138] Furthermore, the implementation steps of the visual inspection invention can also cover both operational procedures and parameter settings. First, the image acquisition module can be activated, and a water-cooled, high-temperature camera with 8 megapixels or higher can be deployed on-site based on the inspection requirements. Once activated, the camera continuously captures image data at a set frame rate and transmits it to the edge computing server.

[0139] In terms of parameter settings, the image acquisition module needs to set parameters such as camera resolution, frame rate, and exposure time to adapt to different detection scenarios. The edge computing server needs to adjust the weight parameters of the fusion algorithm to ensure the characteristics of the target object and set parameters such as the number of neural network layers and neurons to optimize feature extraction. The decision-making module needs to set the judgment threshold to ensure the accuracy and reliability of the detection results. By properly setting these parameters, efficient and accurate visual inspection can be achieved.

[0140] As can be seen from the above description, the glass inspection method of this embodiment overcomes the limitations of surface reflections through multispectral fusion technology, enabling online detection of internal defects in liquid glass for the first time. Its dual-mode selection allows for adaptability to a variety of scenarios. For example, the lightweight model can adapt to complex environments, improving detection accuracy; while the non-network model detection algorithm can adapt to general environments, achieving higher detection speeds and lower latency.

[0141] Secondly, the glass detection processing method of this embodiment has both high precision and low latency, such as lightweight model GPU acceleration to meet the needs of high-speed production lines (≥1m / s); the dynamic optical compensation algorithm reduces the impact of environmental temperature drift (accuracy ±0.1℃) and has strong anti-interference ability.

[0142] At the same time, in terms of video recording, the glass detection processing method of this embodiment adopts GPU-based hardware encoding technology, fully utilizing the parallel computing capabilities of the GPU, further reducing the encoding delay, and realizing fast and high-quality preservation of the detection video, providing strong support for subsequent data analysis and problem tracing.

[0143] In addition, the glass detection processing method of this embodiment adopts a multi-threaded parallel processing architecture, which greatly improves the overall operating efficiency, realizes real-time detection, alarm and recording of video streams, and effectively meets the strict requirements of real-time and high efficiency in industrial production.

[0144] Applying the glass inspection processing method of this embodiment to actual production applications can achieve quantitative indicators: defect detection rate ≥ 99.9%, false alarm rate ≤ 0.2%; inspection speed reaches 200 frames / second, supporting inspection of ≥ 5000 points per meter of glass plate; system power consumption is reduced by 40%. Qualitative effects include: identification of 0.1mm 2 The following minor defects; reduce the workload of personnel, reduce the probability of equipment damage, and assist in process optimization.

[0145] Therefore, the glass detection processing method of this embodiment can detect foreign matter in the glass liquid in time and issue an alarm in time through real-time online visual processing at the front end, and manually intervene and process in time according to the system prompts. Early detection and early processing not only improve the continuity of product production and product quality, but also solve the problem of foreign matter in the glass liquid damaging subsequent production equipment, thereby saving production costs.

[0146] Corresponding to the processing method of glass detection in the above embodiment, Figure 7 This is a structural block diagram of a glass detection processing device provided in one embodiment of the present application. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0147] Reference Figure 7 , the device comprises:

[0148] An acquisition module 301 is configured to acquire raw image data of the glass to be inspected in response to a user's inspection instruction for the glass to be inspected, wherein the raw image data includes red, green, and blue image data and / or ultraviolet and visible light image data, and the glass to be inspected is photovoltaic glass in a molten state;

[0149] The feature extraction module 302 is used to extract features from the original image data to obtain target feature data of the glass to be inspected;

[0150] The analysis module 303 is used to analyze the target feature data to obtain a detection result of the glass to be detected. The detection result is used to indicate whether there is foreign matter in the glass to be detected.

[0151] This embodiment provides a glass inspection processing device. The device, through an acquisition module, responds to a user's inspection instruction for the glass to be inspected, and obtains raw image data of the glass to be inspected. The raw image data includes red, green, and blue image data and / or ultraviolet-visible light image data. The glass to be inspected is photovoltaic glass in a molten state. The feature extraction module extracts features from the raw image data to obtain target feature data of the glass to be inspected. The analysis module analyzes the target feature data to obtain an inspection result of the glass to be inspected. The inspection result is used to characterize whether there is foreign matter in the glass to be inspected. Using this device, by extracting features from the red, green, and blue image data and / or ultraviolet-visible light image data of the glass to be inspected, the device can penetrate the surface layer of the glass liquid to detect foreign matter in the glass liquid. While avoiding missed detections due to visual fatigue, it improves the efficiency and accuracy of glass inspection, thereby improving product quality.

[0152] Optionally, the detection instruction carries a target algorithm identifier, and the feature extraction module includes:

[0153] a first feature extraction unit configured to, if the target algorithm identifier is detected to be the first algorithm identifier, use an improved YOLO target detection model to perform feature extraction on the original image data to obtain target feature data of the glass to be detected, wherein the image fusion processing module in the improved YOLO target detection model includes multiple transformer networks, and the image fusion processing module is configured to perform fusion processing on the image data;

[0154] The second feature extraction unit is used to extract features from the original image data based on the historical image data of the glass to be detected, if the target algorithm identifier is detected to be the second algorithm identifier, to obtain target feature data of the glass to be detected, wherein the historical image data and the original image data are respectively image data of the glass to be detected at different time nodes.

[0155] Optionally, the original image data includes red, green, and blue image data and ultraviolet and visible light image data, and the first feature extraction unit includes:

[0156] a preprocessing subunit, configured to preprocess the red, green, and blue image data and the ultraviolet and visible light image data respectively to obtain first processed image data and second processed image data, wherein the preprocessing at least includes a cropping process;

[0157] an extraction subunit, configured to perform feature extraction on the first processed image data and the second processed image data using an improved YOLO target detection model to obtain extracted image data;

[0158] The format conversion subunit is used to perform format conversion processing on the extracted image data to obtain target feature data of the glass to be detected. The format conversion processing is used to convert the format of the extracted image data into a preset format.

[0159] Optionally, the extraction subunit is specifically used to:

[0160] Perform convolution processing on the first processed image data and the second processed image data respectively to obtain third processed image data and fourth processed image data;

[0161] inputting the third processed image data and the fourth processed image data into an image fusion processing module to obtain first fused image data and second fused image data, wherein the first processed image data and the first fused image data have the same spectral value, and the second processed image data and the second fused image data have the same spectral value;

[0162] performing an exclusive OR calculation on the third processed image data and the first fused image data to obtain fifth processed image data, and performing an exclusive OR calculation on the fourth processed image data and the second fused image data to obtain sixth processed image data;

[0163] Extracted image data is obtained by calculation based on the fifth processed image data and the sixth processed image data. The extracted image data at least includes a grayscale value.

[0164] Optionally, inputting the third processed image data and the fourth processed image data into an image fusion processing module to obtain first fused image data and second fused image data includes:

[0165] performing splicing processing on the third processed image data and the fourth processed image data to obtain spliced ​​image data;

[0166] Performing an XOR calculation on the spliced ​​image data and the preset position code to obtain first calculation data;

[0167] Inputting the first calculated data into multiple transformer networks in the image fusion processing module in sequence to obtain target fused image data;

[0168] The target fused image data is split to obtain first fused image data and second fused image data.

[0169] Optionally, each transformer network includes a first normalization layer, a multi-head attention mechanism unit, a second normalization layer, and a multi-layer perceptron. The first calculation data is sequentially input into multiple transformer networks in the image fusion processing module to obtain target fused image data, including:

[0170] Normalizing the first calculated data based on the first normalization layer to obtain second calculated data;

[0171] Using a multi-head attention mechanism unit to perform attention mechanism processing on the second calculation data to obtain third calculation data;

[0172] Performing an XOR calculation on the first calculation data and the third calculation data to obtain fourth calculation data;

[0173] Normalizing the fourth calculated data based on the second normalization layer to obtain fifth calculated data;

[0174] Performing perception processing on the fifth calculation data using a multilayer perceptron to obtain sixth calculation data;

[0175] Performing an XOR operation on the fifth calculated data and the sixth calculated data to obtain seventh calculated data;

[0176] The seventh calculated data is used as the first calculated data, and the step of normalizing the first calculated data based on the first normalization layer is returned to obtain the second calculated data, until the first calculated data is input into all transformer networks to obtain the target fused image data.

[0177] Optionally, the original image data includes red, green, and blue image data or ultraviolet and visible light image data, and the target feature data includes the number of target detection areas, and the moving distance and number of successful pixel movements of target pixels within the target detection areas;

[0178] The second feature extraction unit includes:

[0179] Calculate the grayscale difference between the original pixel points at the same position in the original image data and the historical image data to obtain a difference image;

[0180] Merge the original pixels in the difference image according to the grayscale difference of the original pixels in the difference image to generate a candidate detection area of ​​the original image data;

[0181] Position matching is performed on the candidate detection areas of the original image data and the candidate detection areas of the historical image data, and the candidate detection areas with intersecting positions are determined as target detection areas;

[0182] Determine the number of target detection areas, as well as the pixel coordinates and movement distances of target pixels within the target detection areas;

[0183] For each target pixel point in the target detection area, determine whether each target pixel point conforms to the foreign object movement law based on the pixel coordinates of each target pixel point;

[0184] The number of target pixels that conform to the foreign object movement law is determined as the number of successful pixel movements.

[0185] Optionally, the glass detection processing device provided in this embodiment further includes:

[0186] An alarm generation module is used to analyze the target feature data and obtain the detection result of the glass to be detected. If the detection result indicates that there is a foreign object in the glass to be detected, an alarm message of the glass to be detected is generated. The alarm message is used to prompt the user to check the glass to be detected;

[0187] A marking module is used to analyze the target feature data to obtain the detection result of the glass to be inspected, and then mark the foreign matter in the original image data based on the target feature data to obtain marked image data;

[0188] The recording module is used to analyze the target feature data and obtain the detection results of the glass to be detected, and then send a video recording instruction to the image acquisition device. The video recording instruction is used to instruct the image acquisition device to record the video of the glass to be detected.

[0189] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0190] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0191] The embodiment of the present application also provides a terminal device, Figure 8 This is a schematic diagram of the structure of a terminal device provided by an embodiment of the present application. Figure 8 As shown, the terminal device includes: at least one processor 401, a memory 402, an input device 403, an output device 404, and a computer program stored in the memory 402 and executable on at least one processor 401. When the processor 401 executes the computer program, the steps in any of the above-mentioned method embodiments are implemented.

[0192] The input device 403 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the terminal device. The output device 404 may include a display device such as a display screen.

[0193] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor 401, the steps in the above-mentioned method embodiments can be implemented.

[0194] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0195] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor 401, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, a computer-readable storage medium cannot be an electric carrier signal or a telecommunication signal.

[0196] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0197] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0198] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal devices and methods can be implemented in other ways. For example, the device / terminal device embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0199] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0200] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A glass detection processing method, characterized in that: include: In response to a user's inspection instruction for glass to be inspected, obtaining raw image data of the glass to be inspected, wherein the raw image data includes red, green, and blue image data and / or ultraviolet and visible light image data, and the glass to be inspected is photovoltaic glass in a molten state; Performing feature extraction on the original image data to obtain target feature data of the glass to be inspected; The target characteristic data is analyzed to obtain a detection result of the glass to be detected, and the detection result is used to indicate whether there is foreign matter in the glass to be detected.

2. The glass detection processing method according to claim 1, characterized in that: The detection instruction carries a target algorithm identifier, and the feature extraction of the original image data to obtain target feature data of the glass to be detected includes: If it is detected that the target algorithm identifier is the first algorithm identifier, an improved YOLO target detection model is used to perform feature extraction on the original image data to obtain target feature data of the glass to be detected, wherein the image fusion processing module in the improved YOLO target detection model includes multiple transformer networks, and the image fusion processing module is used to perform fusion processing of image data; If it is detected that the target algorithm identifier is the second algorithm identifier, feature extraction is performed on the original image data based on the historical image data of the glass to be detected to obtain target feature data of the glass to be detected, wherein the historical image data and the original image data are respectively image data of the glass to be detected at different time nodes.

3. The glass detection processing method according to claim 2, characterized in that: The original image data includes red, green, and blue image data and ultraviolet and visible light image data. The improved YOLO target detection model is used to perform feature extraction on the original image data to obtain target feature data of the glass to be detected, including: Preprocessing the red, green, and blue image data and the ultraviolet and visible light image data respectively to obtain first processed image data and second processed image data, wherein the preprocessing at least includes cropping; Performing feature extraction on the first processed image data and the second processed image data using an improved YOLO target detection model to obtain extracted image data; The extracted image data is subjected to format conversion processing to obtain target feature data of the glass to be inspected, wherein the format conversion processing is used to convert the format of the extracted image data into a preset format.

4. The glass detection processing method according to claim 3, characterized in that: The adopting of the improved YOLO target detection model to perform feature extraction on the first processed image data and the second processed image data to obtain extracted image data includes: performing convolution processing on the first processed image data and the second processed image data respectively to obtain third processed image data and fourth processed image data; inputting the third processed image data and the fourth processed image data into the image fusion processing module to obtain first fused image data and second fused image data, wherein the first processed image data and the first fused image data have the same spectral value, and the second processed image data and the second fused image data have the same spectral value; performing an exclusive OR calculation on the third processed image data and the first fused image data to obtain fifth processed image data, and performing an exclusive OR calculation on the fourth processed image data and the second fused image data to obtain sixth processed image data; Extracted image data is obtained by calculation based on the fifth processed image data and the sixth processed image data, and the extracted image data at least includes a grayscale value.

5. The glass detection processing method according to claim 4, characterized in that: Inputting the third processed image data and the fourth processed image data into the image fusion processing module to obtain first fused image data and second fused image data includes: performing splicing processing on the third processed image data and the fourth processed image data to obtain spliced ​​image data; Performing an XOR calculation on the spliced ​​image data and a preset position code to obtain first calculation data; Inputting the first calculated data into a plurality of transformer networks in the image fusion processing module in sequence to obtain target fused image data; The target fused image data is split to obtain first fused image data and second fused image data.

6. The glass detection processing method according to claim 5, characterized in that: Each transformer network includes a first normalization layer, a multi-head attention mechanism unit, a second normalization layer, and a multi-layer perceptron. The first calculation data is sequentially input into the multiple transformer networks in the image fusion processing module to obtain target fused image data, including: Normalizing the first calculated data based on the first normalization layer to obtain second calculated data; Using the multi-head attention mechanism unit to perform attention mechanism processing on the second calculation data to obtain third calculation data; Performing an XOR operation on the first calculation data and the third calculation data to obtain fourth calculation data; Normalizing the fourth calculated data based on the second normalization layer to obtain fifth calculated data; Using the multilayer perceptron to perform perceptual processing on the fifth calculation data to obtain sixth calculation data; performing an XOR operation on the fifth calculated data and the sixth calculated data to obtain seventh calculated data; The seventh calculation data is used as the first calculation data, and the step of normalizing the first calculation data based on the first normalization layer to obtain the second calculation data is returned to execute, until the first calculation data is input into all transformer networks to obtain the target fused image data.

7. The glass detection processing method according to claim 2, characterized in that: The original image data includes red, green, and blue image data or ultraviolet and visible light image data, and the target feature data includes the number of target detection areas, and the moving distance and number of successful pixel movements of target pixels within the target detection areas; The step of extracting features from the original image data based on the historical image data of the glass to be inspected to obtain target feature data of the glass to be inspected includes: Calculating the grayscale difference between the original pixel points at the same position in the original image data and the historical image data to obtain a difference image; Merging the original pixels in the difference image according to the grayscale differences of the original pixels in the difference image to generate candidate detection areas of the original image data; Positionally matching the candidate detection areas of the original image data with the candidate detection areas of the historical image data, and determining the candidate detection areas where the positions intersect as target detection areas; Determining the number of target detection areas, and the pixel coordinates and movement distances of target pixels within the target detection areas; For each target pixel point in the target detection area, judging whether each target pixel point conforms to the foreign object movement law based on the pixel coordinates of each target pixel point; The number of target pixels that conform to the foreign object movement law is determined as the number of successful pixel movements.

8. The glass detection processing method according to claim 1, characterized in that: After analyzing the target characteristic data to obtain the detection result of the glass to be detected, the method further includes: If the detection result indicates that there is a foreign object in the glass to be detected, an alarm message of the glass to be detected is generated, and the alarm message is used to prompt the user to check the glass to be detected; Marking the foreign matter in the original image data based on the target feature data to obtain marked image data; A video recording instruction is sent to an image acquisition device, where the video recording instruction is used to instruct the image acquisition device to perform video recording of the glass to be inspected.

9. A glass detection processing device, characterized in that: include: an acquisition module, configured to acquire raw image data of the glass to be inspected in response to a user's inspection instruction for the glass to be inspected, wherein the raw image data includes red, green, and blue image data and / or ultraviolet and visible light image data, and the glass to be inspected is photovoltaic glass in a molten state; A feature extraction module is used to extract features from the original image data to obtain target feature data of the glass to be inspected; The analysis module is used to analyze the target characteristic data to obtain a detection result of the glass to be detected, and the detection result is used to indicate whether there is foreign matter in the glass to be detected.

10. A terminal device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the terminal device implements the method according to any one of claims 1 to 8.

Images