Welding evaluation methods, devices, and storage media based on image data analysis

Abstract

This invention relates to the field of welding technology, and in particular to a welding evaluation method, apparatus, and storage medium based on image data analysis. The method includes the following steps: acquiring a video of the welding process to be evaluated; analyzing and extracting key information from the video; segmenting the video according to the key information and a preset welding process segmentation standard to determine the corresponding operation stage; determining the evaluation information corresponding to the operation stage; by collecting operation videos of the welding site, segmenting the operation videos according to the key information in the videos to determine the corresponding operation stages, and then extracting the evaluation information within the determined operation stages, the evaluator can evaluate the welding operation process based on the evaluation information. This reduces the requirements for the evaluator's concentration, reaction speed, and evaluation proficiency during the evaluation process, and improves the accuracy and reliability of the evaluation results.

Description

[0001] This application is a divisional application of the invention patent application filed on March 7, 2022, with application number 202210223473.2. Technical Field

[0002] This invention relates to the field of welding technology, and in particular to a welding evaluation method, apparatus and storage medium based on image data analysis. Background Technology

[0003] In recent years, with the development of the transportation industry, the demand for railway transportation has been increasing, and railway maintenance has become increasingly important. For cracks and gaps in rails, aluminothermic welding and other techniques are commonly used for rail maintenance and repair. Aluminothermic welding of rails involves applying a sand mold to the rail to be welded, forming a welding cavity; preheating the two joints of the rail within the welding cavity; placing an aluminothermic welding flux, made from aluminothermic welding powder, iron oxide, and other alloying additives in a specific ratio, into a crucible above the welding cavity; igniting the flux; causing an oxidation-reduction reaction of the iron oxide to generate molten aluminothermic steel; within a short time, the high-temperature molten aluminothermic steel melts the self-melting plug at the bottom of the crucible, pouring and filling the welding cavity; the molten aluminothermic steel acts as a filler metal, melting and crystallizing both ends of the rail together; after cooling and solidification, a welded joint is formed; the sand mold is removed, and imperfections are removed and the surface is polished. During rail welding, it is essential to adhere to strict regulations, controlling the timing, extent, and location of each stage, including the preheating, reaction, calming, pouring, demolding, and sprue removal phases, to prevent poor welding quality or even equipment damage. To ensure compliance, each stage of the welding process is typically evaluated. Current evaluation methods involve expert visual observation of the welding process, using experience and a stopwatch to measure the duration and reaction extent of each stage to ensure compliance with welding standards. For example, in the preheating stage, the expert starts the stopwatch immediately after the flame exits the pouring hole and ends it immediately after the flame extinguishes. Close monitoring is crucial throughout the preheating process, especially the color change of the rail web. The preheated rail web reaches approximately 950-1000℃ and turns a bright yellow. In the pouring stage, the timing begins when molten iron flows out of the crucible and ends when no more molten iron flows out.

[0004] This method, in which evaluation experts judge the operation process with their naked eyes, has several drawbacks. It requires a high level of concentration, reaction speed, and evaluation proficiency from the evaluation experts. Furthermore, the evaluation results are significantly influenced by the personal subjective factors of the evaluators, and there may be substantial differences between the evaluation results of different evaluators. Summary of the Invention

[0005] The purpose of this application is to provide a welding evaluation method, device, and storage medium based on image data analysis. By collecting operation videos of the welding site, the operation stage corresponding to the operation video is determined based on the key information in the video, and then the evaluation information within the determined operation stage is extracted, so that the evaluator can evaluate the welding operation process based on the evaluation information. This reduces the requirements for the evaluator's concentration, reaction speed, and evaluation proficiency during the evaluation process, and improves the accuracy and reliability of the evaluation results.

[0006] This application provides a welding evaluation method based on image data analysis, the method comprising the following steps:

[0007] Acquire a video of the welding process to be evaluated; analyze and extract key information from the video; segment the video according to the key information and a preset welding process segmentation standard to determine the operation stage corresponding to the video; determine the evaluation information corresponding to the operation stage.

[0008] Furthermore, the welding evaluation method provided by the present invention also includes:

[0009] The operation stages and corresponding evaluation information are stored and displayed.

[0010] And / or, extract the preset key information evaluation criteria corresponding to the operation stage; compare the evaluation information corresponding to the operation stage with the preset key information evaluation criteria; determine the welding quality of the welding process to be evaluated based on the comparison results.

[0011] Furthermore, the key information includes at least the location and dimensions of the flame and the sand-sealing box; the analysis and extraction of key information from the video specifically includes: extracting an evaluation image from the video; performing preliminary identification on the extracted evaluation image and marking the preliminary identification result in the evaluation image; determining the region of interest in the evaluation image based on the preliminary identification result; processing the preliminary identification result in the region of interest to obtain a process identification marker; analyzing the process identification marker to identify the location and dimensions of the sand-sealing box and the flame.

[0012] Further, determining the region of interest (ROI) in the evaluation image based on the preliminary identification results specifically includes: extracting the sand-sealing box from the preliminary identification results; determining the ROI based on the width, height, and location of the sand-sealing box; the left boundary of the ROI is 1-2 times the width of the sand-sealing box from the left side of the sand-sealing box; the right boundary of the ROI is 1-2 times the width of the sand-sealing box from the right side of the sand-sealing box; the upper boundary of the ROI is 2.3-3.3 times the height of the sand-sealing box from the top edge of the sand-sealing box; and the lower boundary of the ROI is 0.5-1.5 times the height of the sand-sealing box from the bottom edge of the sand-sealing box.

[0013] Further, the processing of the preliminary identification results within the region of interest to obtain process identification markers specifically includes: labeling the preliminary identification results that are completely within the region of interest as the process identification markers; and using the boundary box of the region of interest as a border, cropping a portion of the preliminary identification results that are outside the region of interest, retaining the area of ​​the preliminary identification results within the region of interest, and labeling it as the process identification marker.

[0014] Furthermore, the operation stage is at least one of the following: preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag removal stage; the step of segmenting the video according to the key information and the preset welding process segmentation standard to determine the operation stage corresponding to the video specifically includes: segmenting the video according to the preset welding process, as well as the position and size of the sealing sand box, the flame, and the crucible in the video, to obtain at least one of the following: preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag removal stage.

[0015] Furthermore, the preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag pushing stage are each preset with corresponding reference frame numbers. Each of the preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag pushing stage is equipped with a process identification mark corresponding to each stage, as well as a preset stage label threshold corresponding to each stage. Based on the preset welding sequence, and the position and size of the sealing sand box, as well as the flame and crucible in the video, the video is segmented to obtain at least one of the preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag pushing stage, specifically including:

[0016] Extract consecutive images of reference frames corresponding to the operation stage to be determined from the video images, and use them as determination images; determine whether the total number of process identification marks corresponding to the operation stage to be determined in each determination image meets the corresponding preset stage label threshold; when the total number of process identification marks in the determination images is at the stage threshold corresponding to the operation stage to be determined, update the determination images frame by frame from front to back until the total number of process identification marks in each determination image is greater than the preset stage label threshold for the first time, and determine the time corresponding to this set of determination images as the start time of the operation stage to be determined; continue to update the determination images frame by frame from front to back until the total number of process identification marks in each determination image is less than the preset stage label threshold for the last time, and determine the time corresponding to this set of determination images as the end time of the operation stage to be determined;

[0017] or,

[0018] When the total number of process identification markers in the determination image is at the stage threshold corresponding to the operation stage to be determined, the determination image is updated frame by frame from front to back until the total number of process identification markers in each determination image is less than the preset stage label threshold. At this point, the time corresponding to the set of determination images is determined as the end time of the operation stage to be determined. The determination image is then updated frame by frame from back to front until the total number of process identification markers in each determination image is greater than the preset stage label threshold. At this point, the time corresponding to the set of determination images is determined as the start time of the operation stage to be determined.

[0019] Furthermore, when the operation stage is at least two of the following: preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and sprue pushing stage; for two adjacent operation stages, if the end time of the previous operation stage is later than the start time of the next stage, the time corresponding to the set of determination images that is earlier than the start time of the next stage and is closest to the start time of the next operation stage is determined as the end time of the previous operation.

[0020] Another aspect of this application provides a welding evaluation apparatus based on image data analysis, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the preceding claims.

[0021] In another aspect, this application provides a computer-readable storage medium storing a computer program, characterized in that the program, when executed by a processor, implements the method described in any of the preceding claims.

[0022] 1. The welding evaluation method, apparatus, and computer-readable storage medium based on image data analysis provided in this application employ the following methods: acquiring video of the welding process to be evaluated; analyzing and extracting key information from the video; segmenting the video according to the key information and a preset welding process segmentation standard to determine the corresponding operation stage; and then designing the evaluation information corresponding to the operation stage. By collecting and storing the welding process through on-site operation video, a calculation basis is provided for extracting the operation time and the degree of each operation stage. By analyzing the video to determine the key information in the video that can be used to segment the operation process, and then determining the corresponding operation stage based on the key information, the operation stages of the welding process are automatically segmented, reducing the requirements for the evaluator's concentration, reaction speed, and evaluation proficiency during the evaluation process. Furthermore, the evaluation information, such as the operation points and operation time contained in each operation stage, is automatically extracted, allowing the evaluator to compare the evaluation information with the evaluation standard, thereby improving the accuracy and reliability of the evaluation results.

[0023] 2. The welding evaluation method, apparatus, and computer-readable storage medium based on image data analysis provided in this application further include storing and displaying the operation stage and corresponding evaluation information; and / or extracting preset key information evaluation standards corresponding to the operation stage; comparing the evaluation information corresponding to the operation stage with the preset key information evaluation standards; and determining the welding quality of the welding process to be evaluated based on the comparison results. Storing and displaying the operation stage and corresponding evaluation information allows evaluators to view it at any time, eliminating the need for on-site analysis at the welding operation site, thus increasing the flexibility and convenience of the evaluation work. Comparing the evaluation information corresponding to the operation stage with the preset key information evaluation standards enables automatic evaluation of the welding process to be evaluated.

[0024] 3. The welding evaluation method, apparatus, and computer-readable storage medium based on image data analysis provided in this application utilize key information including at least the position and dimensions of the flame and the sealing sand box. The analysis and extraction of key information from the video specifically includes: extracting evaluation images from the video; performing preliminary identification on the extracted evaluation images and marking the preliminary identification results in the evaluation images; determining the region of interest (ROI) in the evaluation images based on the preliminary identification results; processing the preliminary identification results within the ROI to obtain process identification markers; and analyzing the process identification markers to identify the design of the position and dimensions of the flame and the sealing sand box. By performing image recognition and analysis on the video, the range of the welding operation in the image is determined, the influence of non-welding areas in the image on key information is eliminated, and the amount of data that needs to be processed during key information recognition is reduced, thereby improving the speed of key information recognition and the effectiveness of the recognition results.

[0025] 4. The welding evaluation method, apparatus, and computer-readable storage medium based on image data analysis provided in this application, which uses the determination of the region of interest (ROI) in the evaluation image based on the preliminary identification results, specifically includes: extracting the sealing sand box from the preliminary identification results; determining the ROI based on the width, height, and location of the sealing sand box; the left boundary of the ROI is 1-2 times the width of the sealing sand box from the left side of the sealing sand box, the right boundary of the ROI is 1-2 times the width of the sealing sand box from the right side of the sealing sand box, the upper boundary of the ROI is 2.3-3.3 times the height of the sealing sand box from the top edge of the sealing sand box, and the lower boundary of the ROI is 0.5-1.5 times the height of the sealing sand box from the bottom edge of the sealing sand box. By standardizing the determination criteria of the ROI according to the changing regions in the images corresponding to each operation stage, the standardization of this evaluation method is improved.

[0026] 5. The welding evaluation method, apparatus, and computer-readable storage medium based on image data analysis provided in this application, which processes the preliminary identification results within the region of interest to obtain process identification markers, specifically includes: labeling the preliminary identification results completely within the region of interest as the process identification markers; and cropping a portion of the preliminary identification results outside the region of interest using the bounding box of the region of interest as a border, retaining the area of ​​the preliminary identification results within the region of interest, and labeling it as the process identification marker. Using the region of interest to filter out image changes not caused by welding operations ensures that the extracted key information is generated by the welding operation, improving the credibility of the evaluation information.

[0027] 6. The welding evaluation method, apparatus, and computer-readable storage medium based on image data analysis provided in this application segment the video according to the key information and a preset welding process segmentation standard to determine the corresponding operation stage of the video. Specifically, this includes segmenting the video according to a preset welding sequence flow and the position and size of the flame, crucible, and sealing sand box in the video to obtain at least one design of a preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag removal stage. When determining the operation flow and extracting the evaluation information corresponding to the operation flow, only one operation stage can be extracted, or multiple operation stages can be extracted. The processed video can be the entire welding process or a part of the welding process, which improves the flexibility and practicality of this evaluation method.

[0028] 7. The welding evaluation method, apparatus, and computer-readable storage medium based on image data analysis provided in this application employ a pre-set reference frame number for each of the preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag pushing stage. Each of the preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag pushing stage is equipped with a process identification mark corresponding to each stage, as well as a pre-set stage label threshold design corresponding to each stage. The process identification mark and pre-set stage label threshold are different for each stage, providing a reasonable basis for effectively dividing the operation stages. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a schematic flowchart of one embodiment of the welding evaluation method based on image data analysis provided in this application;

[0031] Figure 2 This is a schematic diagram of key information from an evaluation image in the welding evaluation method based on image data analysis provided in this application;

[0032] Figure 3 This is a schematic diagram of key information from another evaluation image in the welding evaluation method based on image data analysis provided in this application;

[0033] Figure 4 This is a flowchart illustrating another embodiment of the welding evaluation method based on image data analysis provided in this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0035] Figure 1This is a flowchart illustrating one embodiment of a welding evaluation method based on image data analysis as described in this application. While this application provides method operation steps as shown in the following embodiments or accompanying drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order described in the embodiments and accompanying drawings of this application. When the method is applied in actual devices or end products, it can be executed sequentially or in parallel according to the methods shown in the embodiments or accompanying drawings (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed processing environment).

[0036] The welding evaluation method based on image data analysis provided in this application records the welding operation process via video, then provides scientific video analysis standards to segment the operation process corresponding to the video, and then extracts the evaluation information corresponding to each operation process, allowing the operator to directly compare the evaluation information with the evaluation standards. This method concretizes and quantifies evaluation indicators that previously relied solely on human estimation and timing, reducing the requirements for the evaluator's concentration, reaction speed, understanding of the welding process, and evaluation proficiency, while simultaneously improving the reliability of the evaluation results.

[0037] Specifically, such as Figure 1 As shown in the figure, one embodiment of this application provides a welding evaluation method based on image data analysis, the method comprising the following steps:

[0038] S11: Obtain video of the welding process to be evaluated;

[0039] S12: Analyze and extract key information from the video;

[0040] S13: The video is segmented according to the key information and the preset welding process segmentation standard to determine the operation stage corresponding to the video;

[0041] S14: Determine the evaluation information corresponding to the operation stage;

[0042] S15: Store and display the operation stage and the corresponding evaluation information.

[0043] In this embodiment, the key information includes at least the location and dimensions of the sealing sand box, as well as the flame and crucible; the analysis and extraction of key information from the video specifically includes:

[0044] S1201: Extract the evaluation image from the video;

[0045] S1202: Perform preliminary identification on the extracted evaluation image and mark the preliminary identification result in the evaluation image;

[0046] S1203: Determine the region of interest in the evaluation image based on the preliminary identification results;

[0047] S1204: Process the preliminary identification results within the region of interest to obtain a process identification marker;

[0048] S1205: Analyze the process identification markers to identify the location and size of the sealing sand box, as well as the flame and crucible.

[0049] In this embodiment, determining the region of interest in the evaluation image based on the preliminary recognition result specifically includes:

[0050] Extract the sealing sandbox from the preliminary identification results;

[0051] The region of interest is determined based on the width, height, and location of the sand sealing box. The left boundary of the region of interest is 1 to 2 times the width of the sand sealing box from the left side of the sand sealing box; the right boundary of the region of interest is 1 to 2 times the width of the sand sealing box from the right side of the sand sealing box; the upper boundary of the region of interest is 2.3 to 3.3 times the height of the sand sealing box from the top edge of the sand sealing box; and the lower boundary of the region of interest is 0.5 to 1.5 times the height of the sand sealing box from the bottom edge of the sand sealing box.

[0052] like Figure 2 The diagram shown is a schematic representation of the process identification markers used to identify the preheating stage in the welding evaluation method based on image data analysis provided in this application. Figure 3 The diagram shown is a schematic representation of the process identification markers used to identify the reaction stage in the welding evaluation method based on image data analysis provided in this application. To identify the location of the region of interest in the image, the attached diagram... Figure 2 and appendix Figure 3 The area outside the region of interest was not cropped out.

[0053] like Figure 2 , Figure 3 As shown, in this embodiment, the left boundary of the region of interest 110 is one width of the sand sealing box from the left side of the sand sealing box; the right boundary of the region of interest is one width of the sand sealing box from the right side of the sand sealing box; the upper boundary of the region of interest is 2.3 times the height of the top of the sand sealing box from the top of the sand sealing box; and the lower boundary of the region of interest is 0.5 times the height of the bottom of the sand sealing box from the bottom of the sand sealing box.

[0054] In this embodiment, the processing of the preliminary identification results within the region of interest to obtain a process identification marker specifically includes:

[0055] The preliminary identification results that are entirely within the region of interest (ROI) are marked as process identification markers. Using the bounding box of the ROI as a border, a portion of the preliminary identification results outside the ROI are cropped, retaining only the areas within the ROI, which are then marked as process identification markers. From all the preliminary identification results, those completely outside the ROI are removed, as are portions of the preliminary identification results that cross the ROI bounding box. This application of the ROI to filter the preliminary identification results removes areas in the image unrelated to welding, effectively reducing the computational load for subsequent determination of key information and lowering the computational and storage requirements of the hardware.

[0056] In this embodiment, the operation stages are preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag removal stage; the step of segmenting the video according to the key information and the preset welding process segmentation standard to determine the operation stage corresponding to the video specifically includes:

[0057] Based on the preset welding sequence, as well as the position and size of the sealing sand box, flame, and crucible in the video, the video is segmented to obtain the preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag pushing stage.

[0058] In this embodiment, the preheating stage, reaction stage, calming stage, casting stage, demolding stage, and sprue pushing stage are each preset with a corresponding number of reference frames. The number of reference frames for each stage is usually 20-40 frames. In this embodiment, the number of reference frames for each stage is 20 frames. The preheating stage, reaction stage, calming stage, casting stage, demolding stage, and sprue pushing stage are each equipped with a process identification mark corresponding to each stage, as well as a preset stage label threshold corresponding to each stage.

[0059] Based on the preset welding process, and the position and size of the sealing sand box, as well as the flame and crucible in the video, the video is segmented to obtain at least one of the following stages: preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag removal stage, specifically including:

[0060] Extract consecutive images from the video that correspond to the reference frame number of the operation stage to be determined, and use them as the determination images;

[0061] Determine whether the total number of process identification tags corresponding to the operation stage to be determined in each of the determination images meets the corresponding preset stage tag threshold.

[0062] When the total number of process identification markers in the judgment image is less than the stage threshold corresponding to the operation stage to be determined, the judgment image is updated frame by frame from front to back until the total number of process identification markers in each judgment image is greater than the preset stage label threshold for the first time. At this time, the time corresponding to the group of judgment images is determined as the start time of the operation stage to be determined. The judgment image is updated frame by frame from front to back until the total number of process identification markers in each judgment image is less than the preset stage label threshold for the last time. At this time, the time corresponding to the group of judgment images is determined as the end time of the operation stage to be determined.

[0063] or,

[0064] When the total number of process identification markers in the determination image is at the stage threshold corresponding to the operation stage to be determined, the determination image is updated frame by frame from front to back until the total number of process identification markers in each determination image is less than the preset stage label threshold. At this point, the time corresponding to the set of determination images is determined as the end time of the operation stage to be determined. The determination image is then updated frame by frame from back to front until the total number of process identification markers in each determination image is greater than the preset stage label threshold. At this point, the time corresponding to the set of determination images is determined as the start time of the operation stage to be determined.

[0065] like Figure 2-3 As shown, in this embodiment, the process identification markers corresponding to the preheating stage are: preheating flame 101, preheating frame 102, back spray gun, front spray gun and hand, and back spray gun and hand; the corresponding preset stage label threshold is 2-15; wherein the preheating flame is a flame located on both sides of the sandbox (e.g., Figure 2 As shown in Figure 101, it is identified from the image based on the coordinates of the sandbox in the image and the shape of the flame, and marked with a box in the image; the preheating frame (such as...) Figure 2As shown in Figure 102, a frame with a preset size is used to enclose two preheating flames, a sealing sand box, and an ash tray. The intersection-union ratio of the sum of the identification frames for the preheating flame, the sealing sand box, and the ash tray with the preheating frame is typically 1 / 2. The process identification markers corresponding to the reaction stage are: fire 103 on the crucible lid (greater than a first area threshold), crucible lid and fire 104, and reaction frame 105; the corresponding preset stage label threshold is 2-15. The crucible lid can be anchored according to the size, position, and shape of the sealing sand box, and the fire on the crucible lid, the crucible lid, and the fire can also be identified and marked in the image. The reaction frame is a frame with a preset size that can enclose the crucible lid, the fire on the crucible lid, and the sealing sand box. The intersection-union ratio of the sum of the frames for the crucible lid, the fire on the crucible lid, and the sealing sand box with the reaction frame is typically 1 / 2.

[0066] The process identification marker corresponding to the calm phase is the fire on the crucible lid that is smaller than the first area threshold; the corresponding preset phase label threshold is 2-15. Compared to the reaction phase, the fire on the top of the crucible lid is significantly smaller, so it can be determined whether it is "the fire on the crucible lid that is smaller than the first area threshold" based on the area occupied by the fire on the crucible lid in the sensing interest area. The area of ​​the fire on the crucible lid is generally determined by calculating the number of pixels it occupies. For ease of calculation, the number of pixels occupied by the frame of the fire on the crucible lid can be directly calculated. The size of the first area threshold is usually related to the image clarity in the video, the distance between the shooting lens and the sealing sand box, that is, the position of the sealing sand box in the image, etc. Since the flame in the calm phase is significantly smaller than the flame in the reaction phase, the phase corresponding to the flame can be judged by the naked eye. Those skilled in the art can preset the first area threshold based on experience, or they can extract one frame of the reaction phase image and one frame of the calm phase image before starting the evaluation and calculate the first area threshold. In summary, there are multiple methods to determine the first area threshold, and those skilled in the art can choose according to actual needs, which will not be elaborated here.

[0067] The process identification markers corresponding to the casting stage are: the flame in the gap between the crucible and the sealing sand box, the flame during the casting stage, flowing molten iron, stationary molten iron, the crucible and the casting flame, and the casting frame; the corresponding preset stage label threshold is 2-15; among them, the flame in the gap between the crucible and the sealing sand box can be identified and framed according to the position of the crucible and the sealing sand box in the image and the shape of the flame; flames will emerge from both sides below the crucible, so the flame at this position is the flame of the casting stage, which can be identified and framed in the image; after the calming stage, molten iron will flow out from the side and drip, so flowing molten iron and stationary molten iron will appear, which can be identified and framed in the image; the identification method for the casting frame is the same as the identification method for the preheating frame, and the intersection-union ratio of the frame of the casting stage flame and the frame of the crucible and the casting flame is 1 / 2.

[0068] The process identification mark corresponding to the demolding stage is: mold clamping plate; the corresponding preset stage label threshold is 1-10.

[0069] The process identification markers corresponding to the tumor pushing stage are: front of the tumor pushing machine; back of the tumor pushing machine; left of the joystick; right of the joystick; and the corresponding preset stage label threshold is 1-10.

[0070] In this embodiment, the YOLOv3-Tiny model is used to identify the labels in the above process. The basic network of this model is the Darknet-53 convolutional neural network. The YOLOv3-Tiny model is existing technology. During training, each image is processed to a 720P screen size. After multiple training sessions, the above labels in each image can be identified. Applying the YOLOv3-Tiny model to identify specific shapes after training is existing technology and will not be elaborated further here.

[0071] In this embodiment, determining the evaluation information corresponding to the operation stage specifically includes:

[0072] The duration of the preheating stage is calculated based on the start and end times of the preheating stage.

[0073] The duration of each reaction stage is calculated based on the start and end times of the reaction stage.

[0074] The duration of the calm phase is calculated based on the start and end times of the calm phase.

[0075] The duration of the pouring process is calculated based on the start and end times of the pouring stage.

[0076] The time from the end of the pouring stage to the demolding stage is calculated based on the end time of the pouring stage and the start time of the demolding stage.

[0077] The time from the end of the pouring stage to the start of the tumour pushing stage is calculated based on the end time of the pouring stage and the start time of the tumour pushing stage.

[0078] The evaluation information described in this embodiment may also include the start and end times of each of the above stages, so that operators can retrieve the video of the corresponding operation stage according to the start and end times, observe the operation process through the video, and then observe each operation in the welding process. Example 2

[0079] Another embodiment of this application provides a welding evaluation method based on image data analysis, such as... Figure 4 As shown, this embodiment is an improvement based on Embodiment 1. The parts already described in Embodiment 1 will not be repeated here.

[0080] This embodiment provides a welding evaluation method based on image data analysis, which includes the following steps:

[0081] S21: Obtain video of the welding process to be evaluated;

[0082] S22: Analyze and extract key information from the video;

[0083] S23: The video is segmented according to the key information and the preset welding process segmentation standard to determine the operation stage corresponding to the video;

[0084] S24: Determine the evaluation information corresponding to the operation stage;

[0085] S25: Extract the preset key information evaluation criteria corresponding to the operation stage;

[0086] S26: Compare the evaluation information corresponding to the operation stage with the preset key information evaluation criteria;

[0087] S27: Determine the welding quality of the welding process to be evaluated based on the comparison results.

[0088] In this embodiment, the left boundary of the region of interest is 2 times the width of the sand sealing box from the left side of the sand sealing box; the right boundary of the region of interest is 2 times the width of the sand sealing box from the right side of the sand sealing box; the upper boundary of the region of interest is 3.3 times the height of the top of the sand sealing box from the top of the sand sealing box; and the lower boundary of the region of interest is 1.5 times the height of the bottom of the sand sealing box from the bottom of the sand sealing box.

[0089] In this embodiment, the operation stage is one of the following: preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and sprue removal stage; each preheating stage has corresponding evaluation information and preset key information evaluation standards.

[0090] In this embodiment, determining the evaluation information corresponding to the operation stage specifically includes:

[0091] The duration of the preheating stage is calculated based on the start and end times of the preheating stage.

[0092] The duration of each reaction stage is calculated based on the start and end times of the reaction stage.

[0093] The duration of the calm phase is calculated based on the start and end times of the calm phase.

[0094] The duration of the pouring process is calculated based on the start and end times of the pouring stage.

[0095] The time from the end of the pouring stage to the demolding stage is calculated based on the end time of the pouring stage and the start time of the demolding stage.

[0096] The time from the end of the pouring stage to the start of the tumour pushing stage is calculated based on the end time of the pouring stage and the start time of the tumour pushing stage.

[0097] The preset key information evaluation standard for the preheating stage is whether the duration is between 3 minutes and 30 seconds and 4 minutes and 30 seconds.

[0098] The preset key information evaluation standard for the reaction phase is whether the duration is within 7-15 seconds;

[0099] The preset key information evaluation criteria for the calm phase are whether the duration is between 6 and 18 seconds; and whether the total time of the reaction phase and the calm phase is no more than 35 seconds.

[0100] The preset key information evaluation standard for the pouring stage is whether the duration is within 10-600 seconds;

[0101] The preset key information evaluation standard for the demolding stage is whether the time from the end of pouring to demolding meets 4 minutes and 30 seconds.

[0102] The preset key information evaluation standard for the tumor pushing stage is whether the time from the end of pouring to tumor pushing meets 7 minutes and 30 seconds.

[0103] In this embodiment, determining the welding quality of the welding process to be evaluated based on the comparison results specifically means that when the duration of the operation stage being evaluated meets the corresponding preset key information evaluation standard, it is determined to be qualified; otherwise, it is determined to be unqualified.

[0104] Preferably, in this embodiment, the operation stage to be determined is the preheating stage. The reference frame number is 40 frames, and the process identification markers are preheating flame, preheating frame, back spray gun, front spray gun and hand, and back spray gun and hand; the corresponding preset stage label threshold is 2-15.

[0105] The step of segmenting the video according to the key information and the preset welding process segmentation standard to determine the preheating stage corresponding to the video specifically includes:

[0106] Extract consecutive images from the video corresponding to the reference frame number of the preheating stage, and use them as the determination images;

[0107] Determine whether the total number of process identification tags corresponding to the preheating stage in each of the determination images meets the corresponding preset stage tag threshold;

[0108] When the total number of process identification markers in the judgment image is at the threshold corresponding to the preheating stage, the judgment image is updated frame by frame from front to back until the total number of process identification markers in each judgment image is greater than the preset stage label threshold for the first time. At this time, the time corresponding to this group of judgment images is determined as the start time of the preheating stage. The judgment image is updated frame by frame from front to back until the total number of process identification markers in each judgment image is less than the preset stage label threshold for the last time. At this time, the time corresponding to this group of judgment images is determined as the end time of the preheating stage.

[0109] or,

[0110] When the total number of process identification markers in the determination image is at the threshold corresponding to the preheating stage, the determination image is updated frame by frame from front to back until the total number of process identification markers in each determination image is less than the preset stage label threshold. At this point, the time corresponding to the set of determination images is determined as the end time of the preheating stage. The determination image is then updated frame by frame from back to front until the total number of process identification markers in each determination image is greater than the preset stage label threshold. At this point, the time corresponding to the set of determination images is determined as the start time of the preheating stage.

[0111] When calculating the total number of process identification tags, one possible method is to count the number of times one of the predicted tags appears in an image as 1, the number of times N predicted tags appear as N, and the number of times one predicted tag appears in M ​​images as M. The total number of process identification tags is the total number of predicted tags in 40 frames of images. Another method is to assign different weights to different process identification tags based on the uniqueness of each tag, and then calculate the total number. For example, the weight coefficient for preheating flames is a1, the weight coefficient for preheating large frames is a2, the weight coefficient for the back spray gun is a3, the weight coefficient for the front spray gun and hand is a4, and the weight coefficient for the back spray gun and hand is a5. In this case, the total number of process identification tags in the judgment image of the reference frame number = the number of preheating flames * a1 + the number of preheating large frames * a2 + the number of back spray guns * a3 + the number of front spray guns and hands * a4 + the number of back spray guns and hands * a5. Similarly, the total number of process identification markers in the judgment image corresponding to the reference frame number during the reaction stage = the number of flames on the crucible lid greater than the first area threshold * b1 + the number of crucible lids and flames * b2 + the number of reaction frames * b3. The total number of process identification markers in the judgment image corresponding to the casting stage = the number of flames in the gap between the crucible and the sealing sand box * c1 + the number of flames in the casting stage * c2 + the number of flowing molten iron * c3 + the number of stationary molten iron * c4 + the number of crucibles and casting flames * c5 + the number of casting frames * c6. The total number of process identification markers in the judgment image corresponding to the lumping stage = the number on the front of the lumping machine * d1 + the number on the back of the lumping machine * d2 + the number on the left side of the joystick * d3 + the number on the right side of the joystick * d4. In this calculation method, the preset stage label thresholds for each stage remain unchanged. Example 3

[0112] This embodiment is a technical solution similar to Embodiment 1 or Embodiment 2. The biggest difference between this embodiment and Embodiment 1 or Embodiment 2 is that the operation stages to be determined in this embodiment are at least two of the following: preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and sprue pushing stage.

[0113] When the operation phases that need to be determined include multiple adjacent operation phases,

[0114] For two consecutive operation phases, if the end time of the previous operation phase is later than the start time of the next operation phase, the time corresponding to the set of judgment images that is earlier than the start time of the next operation phase and is closest to the start time of the next operation phase is determined as the end time of the previous operation.

[0115] Preferably, in this embodiment, the video of the welding process to be evaluated includes both the preheating stage and the reaction stage.

[0116] When the end time of the preheating stage is later than the start time of the reaction stage, the time corresponding to the set of judgment images that is earlier than the start time of the reaction stage and is closest to the start time of the reaction stage is determined as the end time of the preheating stage.

[0117] In this embodiment, the left boundary of the region of interest is 1.5 times the width of the sand sealing box from the left side of the sand sealing box; the right boundary of the region of interest is 1.5 times the width of the sand sealing box from the right side of the sand sealing box; the upper boundary of the region of interest is 3 times the height of the top of the sand sealing box from the top of the sand sealing box; and the lower boundary of the region of interest is 1 time the height of the bottom of the sand sealing box from the bottom of the sand sealing box.

[0118] Based on the same inventive concept, embodiments of this application also provide a welding evaluation apparatus based on image data analysis, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the method described in any of the above embodiments. Since the principle of the welding evaluation apparatus based on image data analysis is similar to that of the welding evaluation method based on image data analysis, the implementation of the welding evaluation apparatus based on image data analysis can refer to the implementation of the welding evaluation method based on image data analysis, and repeated details will not be elaborated further. As used below, the terms "unit," "subunit," or "module" can refer to a combination of software and / or hardware that performs a predetermined function.

[0119] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of embodiments one to three.

[0120] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, or it can be embodied in the process of data migration. All or part of this application can be used in a variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0121] Although this application has been described by way of examples, those skilled in the art will know that this application has many variations and modifications without departing from the spirit of this application, and it is intended that the appended claims cover such variations and modifications without departing from the spirit of this application.

Claims

1. A welding evaluation method based on image data analysis, characterized in that, The method includes the following steps: Obtain video of the welding process to be evaluated; Analyze and extract key information from the video; The video is segmented according to the key information and the preset welding process segmentation standard to determine the corresponding operation stage; the operation stage is at least one of the following: preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and sprue pushing stage. Determine the evaluation information corresponding to the operation stage; The preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag pushing stage are each equipped with a corresponding reference frame number, process identification mark, and preset stage label threshold. The process identification mark corresponding to the preheating stage includes at least one of the following: preheating flame, preheating frame, back spray gun, front spray gun and hand, and back spray gun and hand. The process identification mark corresponding to the calming stage includes a flame on the crucible lid smaller than a first area threshold. The process identification mark corresponding to the pouring stage includes at least one of the following: flame in the gap between the crucible and the sealing sand box, flowing molten iron, stationary molten iron, crucible, and pouring flame. The process identification mark corresponding to the demolding stage includes a mold clamp. The step of segmenting the video according to the key information and the preset welding process segmentation standard to determine the corresponding operation stage of the video specifically includes: Extract consecutive images from the video that correspond to the reference frame number of the operation stage to be determined, and use them as the determination images; Determine whether the total number of process identification tags corresponding to the operation stage to be determined in each of the determination images meets the corresponding preset stage tag threshold.

2. The method according to claim 1, characterized in that, Also includes: The operation stages and corresponding evaluation information are stored and displayed. And / or, Extract the preset key information evaluation criteria corresponding to the aforementioned operation stage; The evaluation information corresponding to the operation stage is compared with the preset key information evaluation criteria; The welding quality of the welding process to be evaluated is determined based on the comparison results.

3. The method according to claim 2, characterized in that, The key information includes at least the location and dimensions of the sand-sealing box and the flame; the analysis and extraction of key information from the video specifically includes: Extract evaluation images from the video; The extracted evaluation image is initially identified, and the initial identification result is marked in the evaluation image; Based on the preliminary identification results, the region of interest in the evaluation image is determined; The preliminary identification results within the region of interest are processed to obtain process identification markers; The process identification markers were analyzed to identify the location and size of the sealing sand box and the flame.

4. The method according to claim 3, characterized in that, The step of determining the region of interest in the evaluation image based on the preliminary identification results specifically includes: Extract the sealing sandbox from the preliminary identification results; The region of interest is determined based on the width, height, and location of the sand sealing box. The left boundary of the region of interest is 1 to 2 times the width of the sand sealing box from the left side of the sand sealing box; the right boundary of the region of interest is 1 to 2 times the width of the sand sealing box from the right side of the sand sealing box; the upper boundary of the region of interest is 2.3 to 3.3 times the height of the sand sealing box from the top edge of the sand sealing box; and the lower boundary of the region of interest is 0.5 to 1.5 times the height of the sand sealing box from the bottom edge of the sand sealing box.

5. The method according to claim 4, characterized in that, The preliminary identification results within the region of interest are processed to obtain a process identification marker, specifically including: The preliminary identification results that are completely within the region of interest are marked as the process identification markers; and using the bounding box of the region of interest as a border, a portion of the preliminary identification results that are outside the region of interest are cropped, leaving the preliminary identification result area within the region of interest, which is marked as the process identification marker.

6. The method according to claim 5, characterized in that, The step of segmenting the video according to the key information and the preset welding process segmentation standard to determine the corresponding operation stage of the video specifically includes: Based on the preset welding process, as well as the position and size of the sealing sand box, flame, and crucible in the video, the video is segmented to obtain at least one of the following stages: preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag pushing stage.

7. The method according to claim 6, characterized in that, Based on the preset welding process, and considering the position and size of the sealing sand box, as well as the flame and crucible in the video, the video is segmented to obtain at least one of the following stages: preheating stage, reaction stage, calming stage, pouring stage, demolding stage, and slag removal stage, specifically including: Extract consecutive images from the video that correspond to the reference frame number of the operation stage to be determined, and use them as the determination images; Determine whether the total number of process identification tags corresponding to the operation stage to be determined in each of the determination images meets the corresponding preset stage tag threshold. When the total number of process identification markers in the determination image is at a preset stage label threshold corresponding to the operation stage to be determined, the determination image is updated frame by frame from front to back until the total number of process identification markers in each determination image is greater than the preset stage label threshold for the first time. At this time, the time corresponding to the determination image is determined as the start time of the operation stage to be determined. The determination image is then updated frame by frame from front to back until the total number of process identification markers in each determination image is less than the preset stage label threshold for the last time. At this time, the time corresponding to the determination image is determined as the end time of the operation stage to be determined. or, When the total number of process identification markers in the determination image is at a preset stage label threshold corresponding to the operation stage to be determined, the determination image is updated frame by frame from front to back until the total number of process identification markers in each determination image is less than the preset stage label threshold. At this point, the time corresponding to the determination image is determined as the end time of the operation stage to be determined. The determination image is then updated frame by frame from back to front until the total number of process identification markers in each determination image is greater than the preset stage label threshold. At this point, the time corresponding to the determination image is determined as the start time of the operation stage to be determined.

8. The method according to claim 7, characterized in that, When the operation phase is at least two of the following: preheating phase, reaction phase, calming phase, pouring phase, demolding phase, and sprue pushing phase; For two adjacent operation phases, if the end time of the previous operation phase is later than the start time of the next operation phase, the time corresponding to the set of judgment images that is earlier than the start time of the next operation phase and is closest to the start time of the next operation phase is determined as the end time of the previous operation.

9. A welding evaluation device based on image data analysis, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 8.

Images