Cable head processing progress monitoring method and monitoring device based on image recognition

Abstract

The invention provides a cable head processing progress monitoring method and device based on image recognition, and the method comprises the following steps: collecting first image information which is used for representing the installation progress of a cable head; obtaining preprocessed second image information based on the first image information; determining a first recognition feature and a second recognition feature based on the second image information, the first recognition feature being an article feature, and the second recognition feature being a gesture feature; and determining the current cable head processing progress based on the first identification feature and the second identification feature. According to the cable head processing progress monitoring method and the monitoring device, the cable head manufacturing process can be effectively identified, cable head side station supervision work is promoted to be comprehensive, intelligent and standardized, management of the cable head manufacturing work process is facilitated, and the work efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to the field of cable head processing monitoring technology, and specifically to a cable head processing progress monitoring method and monitoring device based on image recognition. Background Art

[0002] At present, the operation mode of power companies is mainly based on manual operation and manual record keeping. On-site supervision of cable heads is a typical representative of this, which has the disadvantages of numerous tools, cumbersome work, difficulty in comprehensive supervision, low efficiency, and high error rate. In the production of cable heads, supervisors are required to conduct on-site supervision according to the production requirements of cable heads. The supervision method is mainly for supervisors to take photos, check, and record on site. This supervision method has the following disadvantages: it is difficult to supervise the entire work process comprehensively and efficiently; supervisors are manual supervisors, and since carelessness is human nature, a high error rate is inevitable in the supervision process; supervision information is incomplete, and it is impossible to achieve comprehensive performance evaluation and responsibility tracing through supervision information; the supervision results are pictures, and they need to be entered into the database twice, resulting in low work efficiency.

[0003] Therefore, there is a technical problem in the prior art that the efficiency of manual monitoring of the processing progress of the existing cable head is low. Summary of the invention

[0004] The main purpose of the present invention is to provide a cable head processing progress monitoring method and monitoring device based on image recognition, so as to solve the technical problem of low efficiency of existing manual monitoring of cable head processing progress in the prior art.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present invention, there is provided a cable head processing progress monitoring method based on image recognition, comprising the following steps: acquiring first image information, wherein the first image information is used to characterize the installation progress of the cable head; obtaining preprocessed second image information based on the first image information; determining a first recognition feature and a second recognition feature based on the second image information, wherein the first recognition feature is an object feature and the second recognition feature is a gesture feature; and determining the current cable head processing progress based on the first recognition feature and the second recognition feature.

[0006] In some embodiments, obtaining the preprocessed second image information based on the first image information includes sequentially performing damage cleaning, blur cleaning, and similarity cleaning on the first image information to obtain the second image information.

[0007] In some embodiments, determining the first identification feature based on the second image information includes: determining the first identification feature based on the second image information using a yolov8 model.

[0008] In some embodiments, determining a second recognition feature based on the second image information includes: obtaining a cable head installation training sample set and performing category labeling, wherein the category label is used to characterize dynamic gestures; using the labeled training samples as input to construct a gesture recognition model, wherein the gesture recognition model includes a dynamic channel and a static channel, wherein the dynamic channel uses image processing technology and a CNNLSTM neural network to capture the motion trajectory of the gesture; the static channel uses a structural similarity index SSIM to determine the key frames of the gesture to construct a gesture recognition model combining the dynamic channel and the static channel; and determining the second recognition feature based on the dynamic channel and the static channel.

[0009] In some embodiments, determining the second identification feature based on the dynamic channel and the static channel includes: acquiring a dynamic gesture feature based on the dynamic channel; acquiring a static gesture feature based on the static channel; and determining the second identification feature based on the dynamic gesture feature and the static gesture feature.

[0010] In some embodiments, the dynamic gesture feature acquisition based on the dynamic channel includes: performing inter-frame difference on two adjacent frames of images in a video sequence to obtain a foreground image of a gesture motion area, performing difference between the motion area graphic and the background image to extract a motion target image, and comparing the motion target image with a threshold to obtain a binary image; obtaining a segmented gesture foreground video based on the binary image, and inputting it into a CNNLSTM network to extract motion features; the CNNLSTM architecture includes a convolution layer, a maximum pooling layer, a flattening layer, a long short-term memory LSTM layer, and a Softmax output layer, wherein the convolution layer and the maximum pooling layer are connected in parallel, and the input differential image sequence is flattened by the flattening layer and then input into the LSTM layer and the Softmax layer in turn. x output layer, the output of the Softmax output layer is used as the final output of the CNNLSTM architecture; the input differential image sequence first passes through the first convolution layer to extract local features, then passes through the first maximum pooling layer Maxpooling to reduce the size of the feature map while retaining important features, and then passes through the second convolution layer to further extract features; the output of the second maximum pooling layer passes through the flattening layer Flatten to flatten the feature map into a one-dimensional vector and input it to the LSTM layer, which uses past context information for processing to capture long-term dependencies in the time series; the output of the LSTM layer is fused with the features of the static channel output, and passes through the Softmax activation function to generate a probability distribution for each gesture category and assign a gesture label.

[0011] In some embodiments, static gesture features are obtained based on a static channel, including: using a structural similarity index SSIM to calculate the similarity between adjacent frames, and determining the peak moment of the motion gesture by analyzing the change trend of SSIM; after determining the gesture peak frame, removing the background in the gesture peak frame by a background subtraction method; performing YCbCr color space analysis and threshold segmentation on the gesture peak frame image after removing the background to extract the gesture; and locating the position of the hand based on the extracted gesture.

[0012] In some embodiments, determining the second recognition feature based on the dynamic gesture feature and the static gesture feature includes: splicing the output feature vectors extracted from the dynamic channel and the static channel, and inputting the spliced ​​feature vectors into a two-layer fully connected network, using a softmax function to normalize the output to obtain the probability of each gesture category, and selecting the category with the highest probability as the recognition result, wherein the softmax function is defined as follows:

[0013]

[0014] e is the base of natural logarithms, z i is the i-th element of the input vector z, p i is the i-th element of the output vector p, representing the probability of the i-th class.

[0015] In some embodiments, determining the current cable head processing progress based on the first identification feature and the second identification feature includes: comparing the first identification feature and the second identification feature, and if key object information appears in the image frame and a corresponding gesture is performed, the current step can be determined.

[0016] According to another aspect of the present invention, the present invention also provides a cable head processing progress monitoring device based on image recognition, including: an acquisition module, used to acquire first image information, wherein the first image information is used to characterize the installation progress of the cable head; an acquisition module, used to obtain pre-processed second image information based on the first image information; a first determination module; used to determine a first identification feature and a second identification feature based on the second image information, wherein the first identification feature is an object feature and the second identification feature is a gesture feature; a second determination module, used to determine the current cable head processing progress based on the first identification feature and the second identification feature.

[0017] According to another aspect of the present invention, the present invention further provides a storage medium storing a computer program, wherein the computer program implements the steps of any of the aforementioned methods when executed by a processor.

[0018] According to another aspect of the present invention, the present invention also provides an electronic device, comprising at least a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the aforementioned methods when executing the computer program on the memory.

[0019] The cable head processing progress monitoring method and monitoring device provided in the present disclosure are applied to collect first image information, wherein the first image information is used to characterize the installation progress of the cable head; obtain pre-processed second image information based on the first image information; and determine a first recognition feature and a second recognition feature based on the second image information, wherein the first recognition feature is an object feature, and the second recognition feature is a gesture feature; and then determine the current cable head processing progress based on the first recognition feature and the second recognition feature; it can effectively identify the cable head production process, promote the comprehensive, intelligent and standardized on-site supervision of the cable head, facilitate the cable head production workflow management, and improve work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0021] Figure 1 A flowchart of the steps of the cable head processing progress monitoring method based on image recognition provided by the present disclosure is shown;

[0022] Figure 2 A flowchart of steps for obtaining pre-processed second image information based on first image information provided by the present disclosure is shown;

[0023] Figure 3 A flowchart of the steps of determining a second identification feature based on the second image information provided by the present disclosure is shown;

[0024] Figure 4 A flowchart of the steps of determining dynamic gesture features based on dynamic channels provided by the present disclosure is shown;

[0025] Figure 5 A schematic diagram of a flow chart for determining dynamic gesture features based on a dynamic channel provided by the present disclosure is shown;

[0026] Figure 6 A flowchart of the steps of determining static gesture features based on a static channel provided by the present disclosure is shown;

[0027] Figure 7 A schematic diagram of a process for determining static gesture features based on a static channel provided by the present disclosure is shown;

[0028] Figure 8The structure block diagram of the cable head processing progress monitoring device provided by the present disclosure is shown;

[0029] Fig. 9 A schematic structural diagram of an electronic device provided by the present disclosure is shown. DETAILED DESCRIPTION

[0030] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but are not intended to limit the present disclosure.

[0031] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the above description should not be considered as limiting, but merely as an example of an embodiment. Other modifications within the scope and spirit of the present disclosure will occur to those skilled in the art.

[0032] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the general description of the present disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the present disclosure.

[0033] These and other characteristics of the present disclosure will become apparent from the following description of a preferred form of embodiment given as a non-limiting example, with reference to the attached drawings.

[0034] It should also be understood that, although the present disclosure has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the present disclosure that have the features described in the claims and are therefore within the scope of protection defined thereby.

[0035] The above and other aspects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0036] Specific embodiments of the present disclosure are described hereinafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present disclosure, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid obscuring the present disclosure with unnecessary or redundant details. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of ways with substantially any suitable detailed structure.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0038] This specification may use the phrases "in one embodiment," "in another embodiment," "in a further embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present disclosure.

[0039] The present disclosure is further described below in conjunction with the accompanying drawings and specific embodiments.

[0040] In order to solve the technical problem of low efficiency of manual monitoring of cable head processing progress in the prior art, the present invention provides a cable head processing progress monitoring method and monitoring device based on image recognition.

[0041] The cable head processing process includes several different steps. Table 1 shows the main steps involved in the computer processing process, as follows:

[0042] Table 1 Classification of cable intermediate connection installation steps

[0043]

[0044] Specifically, the main steps include the following categories:

[0045] Step 1: Cable pretreatment: The operator begins to strip the outer insulation of the cable and retain a certain amount of steel armor, which is tied and fixed with copper wire, and the rest is stripped; retain a certain amount of inner sheath from the steel armor break, and strip off the remaining inner sheath and filler; from the top of the core wire downward, according to the different specifications of the cable, use PVC tape to mark and strip off the corresponding length of copper shielding; measure the outer semi-conductive layer from the top of the core wire downward, mark and strip off the measured semi-conductive layer with PVC tape according to the cable specifications, and wrap several layers of tape on the insulation layer to protect the insulation layer; after sanding the groove smooth with sandpaper, remove the PVC tape; from the end of each phase core wire downward, remove half the length of the insulation layer of the middle connecting tube to expose the conductor; chamfer the insulation layer at the break and polish it smooth with sandpaper; wipe the insulation layer and copper conductor of each phase with cable cleaning paper, and strip the other end of the cable according to the same size.

[0046] Step 2: Insert the cold shrink joint body: Insert the cold shrink joint body into each phase of the stripped long end of the cable, and insert the copper shielding net into each phase of the short end.

[0047] Step 3: Crimping the intermediate connecting pipe: Use crimping pliers to crimp the intermediate connecting pipes of each phase as required, and polish the connecting pipes to make them smooth and clean. Mark a certain distance inward from the outer semi-conductor break at both ends with PVC tape as the shrink positioning point.

[0048] Step 4: Shrink the body of the cold shrink joint: clean the surface of the insulation layer and evenly apply silicone grease, align the cold shrink joint with the shrink positioning point, and pull out the support bar to allow the joint to shrink naturally.

[0049] Step 5: Install the copper shielding mesh and ground wire: Pull the copper shielding mesh sleeve open and put it on the outside of each phase connector. Use sandpaper to polish the copper shielding layer, insert the ground wire into the bifurcation of the three-core cable, wrap it around the three-phase copper shielding layer for one week and then lead it out. Use a constant force spring to tie the ground wire and the copper shielding mesh sleeve to the copper shielding layer of the three-core cable. Pull the other end of the ground wire to the short end and tie it in the same way. Finally, wrap two layers of PVC tape on the constant force spring.

[0050] Step 6: Internal shaping: Backfill the filler to fill the concave part and make the joint neat. Wrap it tightly with transparent PVC tape. Wrap two layers of waterproof tape between the inner sheaths of the two ends of the cable half-overlappingly and stretch them until they turn pale white.

[0051] Step 7: Connect the external ground wire: Use saw blades and sandpaper to polish the steel armor and remove the anti-rust paint. Use a constant force spring to fix the ground wire to one end of the steel armor and wrap it around the waterproof tape to fix it to the other end of the steel armor. Wrap two layers of PVC tape on the constant force spring. Then overlap the outer sheaths of the cables at both ends and wrap them with two layers of waterproof glue.

[0052] Step 8: Install the armor belt and restore the outer sheath: Wear latex gloves and open the outer packaging of the armor belt. Start from one end and wrap the armor belt half-overlappingly to the other end, then wrap it back until it is used up.

[0053] It can be seen from the above steps that the tools and dynamic gestures used in each step are different, so the processing progress of the cable head can be judged according to the tools and dynamic gestures used.

[0054] Figure 1 The flowchart of the cable head processing progress monitoring method based on image recognition is shown as follows: Figure 1 As shown, the cable head processing progress monitoring method based on image recognition includes the following steps:

[0055] S101, collecting first image information, wherein the first image information is used to represent the installation progress of the cable head.

[0056] In this step, first image information is collected, wherein the first image information is used to characterize the installation progress of the cable head.

[0057] Specifically, the cable head installation and manufacturing process image video is obtained through the image acquisition module, and the cable head installation and manufacturing process image samples are obtained in a frame separation manner. The cable head installation and manufacturing process image video includes different environment image videos, wherein the image acquisition module may, for example, include a high-definition camera.

[0058] S102: Obtain preprocessed second image information based on the first image information.

[0059] After completing the above step S101, in this step, pre-processed second image information is obtained based on the first image information.

[0060] Further, Figure 2 FIG. 4 shows a flow chart of steps of obtaining pre-processed second image information based on the first image information. Figure 2 As shown, obtaining the preprocessed second image information based on the first image information includes sequentially performing damage cleaning, blur cleaning and similarity cleaning on the first image information to obtain the second image information.

[0061] Among them, damage cleaning includes using Python and PIL libraries to detect and count damaged images, and remove damaged images from the image dataset; specifically, when an exception is thrown during the opening process, the image is damaged. All detected damaged images are removed from the image dataset to ensure that the image files in the dataset are available.

[0062] Blur cleaning includes traversing all image data, performing Laplace transform on each image data, and detecting the edges in the image by calculating the second-order derivative of the image. The larger the variance, the more edges and details in the image, and the clearer the image; the smaller the variance, the blurrier the image. Images with a result variance less than a threshold are removed from the image data set to ensure that the image data set is clear; the threshold is set based on the operator's experience.

[0063] Similarity cleaning includes but is not limited to cosine similarity, Euclidean distance, Hamming distance and other measurement methods to calculate the similarity between image feature vectors; images with similarity below a threshold are removed from the data set.

[0064] S103: Determine a first identification feature and a second identification feature based on the second image information, wherein the first identification feature is an object feature, and the second identification feature is a gesture feature.

[0065] After completing the above step S102, in this step, the first identification feature and the second identification feature are determined based on the second image information, wherein the first identification feature is an object feature and the second identification feature is a gesture feature, the object features include a knife, pliers, copper shielding mesh, armor belt, etc., and the gesture features include stripping, inserting, crimping, pulling open, grinding, winding, stretching, etc., so that by determining the first identification feature and the second identification feature, the processing step of the cable can be determined.

[0066] Further, determining the first identification feature and the second identification feature based on the second image information includes determining the first identification feature based on the second image information and determining the second identification feature based on the second image information.

[0067] Among them, determining the first recognition feature based on the second image information includes using the yolov8 model to extract key object features, that is, using the yolov8-based target detection method to determine the first recognition feature.

[0068] Figure 3 A flow chart showing the steps of determining the second identification feature based on the second image information is shown. Figure 3 As shown, determining the second identification feature based on the second image information includes the following steps:

[0069] S301, obtaining a cable head installation training sample set and performing category labeling, wherein the category labeling is used to characterize dynamic gestures.

[0070] In this step, a cable head installation training sample set is obtained and categorized. The categorized labels are used to characterize dynamic gestures to ensure that each gesture can be correctly classified and labeled.

[0071] S302, using the labeled training samples as input to construct a gesture recognition model; wherein the gesture recognition model includes a dynamic channel and a static channel, wherein the dynamic channel uses image processing technology and a CNNLSTM neural network to capture the motion trajectory of the gesture; and the static channel uses a structural similarity index SSIM to determine the key frames of the gesture to construct a gesture recognition model that combines the dynamic channel and the static channel.

[0072] After completing the above step S301, in this step, the labeled training samples are used as input to build a gesture recognition model, wherein the gesture recognition model includes a dynamic channel and a static channel. The dynamic channel uses image processing technology and a CNNLSTM neural network to capture the motion trajectory of the gesture; the static channel uses the structural similarity index SSIM to determine the key frame of the gesture to build a gesture recognition model combining the dynamic channel and the static channel.

[0073] S303: Determine a second identification feature based on the dynamic channel and the static channel.

[0074] After completing the above step S302, in this step, a second identification feature is determined based on the dynamic channel and the static channel.

[0075] Further, determining the second recognition feature based on the dynamic channel and the static channel includes: acquiring a dynamic gesture feature based on the dynamic channel, acquiring a static gesture feature based on the static channel, and determining the second recognition feature based on the dynamic gesture feature and the static gesture feature.

[0076] in, Figure 4 , Figure 5 The flowchart and schematic diagram of the steps of determining dynamic gesture features based on the dynamic channel are shown respectively. Figure 4 As shown, determining dynamic gesture features based on dynamic channels includes the following steps:

[0077] S401, performing inter-frame difference on two adjacent frames in a video sequence to obtain a foreground image of a gesture motion region, performing difference between the motion region image and the background image to extract a motion target image, and comparing the motion target image with a threshold to obtain a binary image.

[0078] Specifically, for A k Each pair of adjacent frame images in (i, j) is subjected to differential processing to obtain the frame difference image D k (i,j), where

[0079] D k (i,j)=A k (i,j)-A k-1 (i,j)

[0080] W k (i,j)=|D k (i,j)-B k (i,j)|

[0081]

[0082] Where: A k (i, j) represents the pixel value at position (i, j) in the kth frame; D k (i, j) represents the pixel difference between adjacent frames; W k (i, j) represents the absolute difference between the frame difference image and the background image, which includes the moving target image and other changing areas; Th represents the preset threshold; T(i, j) represents the generation of a binary image, where pixels with a value of 1 represent moving areas and pixels with a value of 0 represent non-moving areas; Bk(i, j) represents the background image, which is relative to the moving area image and refers to the part of the image that is not properly still.

[0083] S402, based on the binary image, a segmented gesture foreground video is obtained, and the video is input into a CNNLSTM network to extract motion features; the CNNLSTM architecture includes a convolution layer, a maximum pooling layer, a flattening layer, a long short-term memory LSTM layer and a Softmax output layer, wherein the convolution layer and the maximum pooling layer are connected in parallel, and the input differential image sequence is flattened by the flattening layer and then input into the LSTM layer and the Softmax output layer in sequence, and the output of the Softmax output layer is used as the final output of the CNNLSTM architecture.

[0084] After completing the above step S401, in this step, a sequence formed by motion areas in all binary images finally formed in the video sequence is selected as the segmented gesture foreground video, and is input into the CNNLSTM network to extract motion features; the CNNLSTM architecture includes a convolution layer, a maximum pooling layer, a flattening layer, a long short-term memory LSTM layer and a Softmax output layer, wherein the convolution layer and the maximum pooling layer are connected in parallel, and the input differential image sequence is flattened by the flattening layer and then input into the LSTM layer and the Softmax output layer in sequence, and the output of the Softmax output layer is used as the final output of the CNNLSTM architecture.

[0085] Among them, the CNNLSTM architecture is expressed as:

[0086] O(x)=LSTM(Flatten(Maxpooling(Conv2(Maxpooling(Conv1(x))))))

[0087] Among them: Conv1 and Conv2 represent convolutional layers, Maxpooling represents the maximum pooling layer, Flatten represents the flattening layer, and LSTM represents the long short-term memory layer.

[0088] S403, the input differential image sequence first passes through the first convolution layer to extract local features, then passes through the first maximum pooling layer Maxpooling to reduce the size of the feature map while retaining important features, and then passes through the second convolution layer to further extract features.

[0089] After completing the above step S402, in this step, the input differential image sequence first passes through the first convolution layer to extract local features, then passes through the first maximum pooling layer Maxpooling to reduce the size of the feature map while retaining important features, and then passes through the second convolution layer to further extract features.

[0090] Specifically, the convolutional layer and pooling layer operations are defined as follows:

[0091]

[0092] in: is the feature map generated by the lth convolution layer; k ij is the convolution kernel obtained through training; is an additive bias term; pooling max is the maximum pooling operation; tanh is the hyperbolic tangent activation function.

[0093] S404, the output of the second maximum pooling layer passes through the flattening layer to flatten the feature map into a one-dimensional vector and input it into the LSTM layer. The LSTM layer uses past context information for processing to capture long-term dependencies in the time series.

[0094] After completing the above step S403, in this step, the output of the second maximum pooling layer passes through the flattening layer to flatten the feature map into a one-dimensional vector and input it to the LSTM layer. The LSTM layer uses past context information for processing to capture long-term dependencies in the time series.

[0095] Among them, the LSTM layer is defined as follows:

[0096] i t =σ(x t W xi +h t-1 W hi +c t-1 W ci +b i ),

[0097] f t =σ(x t W xf +h t-1 W hf +c t-1 W cf +b f ),

[0098] o t =σ(x t W xo +h t-1 W ho +c t W co +b o ),

[0099]

[0100] Where: x t is the input of the current time step t; i t 、f t , o t 、ct 、h t They are input gate, forget gate, output gate, cell state and hidden state; W xi , W xf , W xo , W xc is the weight between the input layer and the input gate, forget gate, output gate and cell state; W hi , W hf , W ho , W hc is the weight between the hidden layer and the input gate, forget gate, output gate and cell state; b i 、b f 、b o 、b c is the bias term; σ is the sigmoid activation function; Represents element-wise multiplication (Hadamard product); W ci is the weight between the memory gate and the input gate, or the weight between the cell state and the input gate; W cf is the weight between the forget gate and the input gate; W co is the weight between the output gate and the cell state, and tanh is the hyperbolic tangent activation function.

[0101] S405, the output of the LSTM layer is fused with the features of the static channel output, and after passing through the Softmax activation function, the probability distribution of each gesture category is generated and a gesture label is assigned.

[0102] After completing the above step S404, in this step, the output of the LSTM layer is fused with the features of the static channel output, and after passing through the Softmax activation function, a probability distribution of each gesture category is generated, and a gesture label is assigned.

[0103] Further, Figure 6 , Figure 7 The flowchart and schematic diagram of the steps for determining static gesture features based on the static channel are shown respectively. Figure 5 As shown, determining the static gesture features based on the static channel includes the following steps:

[0104] S501, using the structural similarity index SSIM to calculate the similarity between adjacent frames, and determining the peak moment of the motion gesture by analyzing the change trend of the SSIM.

[0105] In this step, the structural similarity index SSIM is used to calculate the similarity between adjacent frames, and the peak moment of the motion gesture is determined by analyzing the change trend of SSIM. The calculation of SSIM is as follows:

[0106]

[0107] where x and y are spatially local windows of the input frame. x and μ y denote the average intensity of x and y, respectively, and σ x and σ y represents the standard deviation of x and y, σ xy represents the covariance of x and y. N is the number of pixels, and x i and i denote the pixel value at index i at x and y, respectively. C1 and C2 are stability constants to avoid division by zero. K1 and K2 are positive numbers much less than 1, and L is the pixel range, which is 255 for an 8-bit grayscale frame.

[0108] S502: After determining the gesture peak frame, remove the background in the gesture peak frame using a background subtraction method.

[0109] After completing the above step S501, in this step, the background in the gesture peak frame is removed using a background subtraction method.

[0110] Specifically, after determining the gesture peak frame, the background in the gesture peak frame is removed using a background subtraction method, wherein the background subtraction method used in this application includes but is not limited to a mixed Gaussian algorithm, an inter-frame difference method, a ViBe algorithm, etc. The above methods are all prior arts, and this application does not limit which specific method is used.

[0111] S503, performing YCbCr color space analysis and threshold segmentation on the gesture peak frame image after background removal to extract the gesture.

[0112] After completing the above step S502, in this step, the gesture peak frame image after removing the background is subjected to YCbCr color space analysis and threshold segmentation to extract the gesture.

[0113] Specifically, the YCbCr color space decomposes the image color information into two parts: brightness (Y) and chrominance (Cb, Cr), where Y represents the brightness information of the image, and Cb and Cr represent the chrominance information of the image. According to the range of skin color changes under different lighting conditions, the thresholds Cb = [80, 120] and Cr = [133, 173] are set. The values ​​of Cb and Cr are compared with the thresholds respectively, and the pixels that meet the thresholds are set to white, and the pixels that do not meet the thresholds are set to black, so as to distinguish the skin color area from other areas. The YCbCr color space conversion and skin color threshold segmentation formulas are as follows:

[0114]

[0115] Among them, SkinMask(i,j) represents the binary mask value at position (i,j).

[0116] S504: Locate the position of the hand based on the extracted gesture.

[0117] After completing the above step S503, in this step, the position of the hand is located based on the extracted gesture.

[0118] Specifically, based on the extracted gesture area, a connected region marking algorithm is used to mark all connected white areas in the image. Each connected region represents an independent skin color region. Based on the size and position information of the connected regions, connected regions that do not belong to the hand are filtered out. For example, smaller connected regions in the image can be removed, or the largest connected region can be selected based on the position of the hand in the image.

[0119] Then, based on the results of the connected region analysis, the position of the hand is located; the hand area is cropped out from the original image and scaled so that its size meets the requirements of the feature extraction network, and then the CNN network is used for feature extraction.

[0120] Furthermore, determining the second recognition feature based on the dynamic gesture feature and the static gesture feature includes fusing the extracted static feature of the gesture peak frame with the dynamic feature of the gesture.

[0121] Specifically, the extracted static features of the gesture peak frame are fused with the dynamic features of the gesture, wherein feature fusion refers to splicing the output feature vectors extracted from the dynamic channel and the static channel. The spliced ​​feature vector contains the dynamic features and static features of the gesture.

[0122] The concatenated feature vector is input into a two-layer fully connected network, and the output is normalized using the softmax function to obtain the probability of each gesture category. The category with the highest probability is selected as the recognition result. The softmax function is defined as follows:

[0123]

[0124] Where e is the base of natural logarithms, z i is the i-th element of the input vector z, p i is the i-th element of the output vector p, representing the probability of the i-th class.

[0125] S104: Determine the current cable head processing progress based on the first identification feature and the second identification feature.

[0126] After completing the above step S103, in this step, the current cable head processing progress is determined based on the first identification feature and the second identification feature.

[0127] Specifically, the first identification feature and the second identification feature are compared, and if the key object information appears in the image frame and the corresponding gesture is performed, the current step can be determined. For example, when the first identification feature is a crimping pliers and the second identification feature is crimping, it can be determined that the current step is crimping the intermediate connecting tube.

[0128] The cable head processing progress monitoring method provided in the present disclosure is applied, by collecting first image information, wherein the first image information is used to characterize the installation progress of the cable head; obtaining pre-processed second image information based on the first image information; and determining a first recognition feature and a second recognition feature based on the second image information, wherein the first recognition feature is an object feature, and the second recognition feature is a gesture feature; and then determining the current cable head processing progress based on the first recognition feature and the second recognition feature; it can effectively identify the cable head production process, promote the comprehensive, intelligent and standardized on-site supervision of the cable head, facilitate the cable head production workflow management, and improve work efficiency.

[0129] In order to better implement the above method, the second aspect of the present disclosure further provides a cable head processing progress monitoring device based on image recognition, which can be integrated on an electronic device.

[0130] For example, Figure 8 As shown, the monitoring device 200 may include: a collection module 210, an acquisition module 220, a first determination module 230 and a second determination module 240, which are specifically as follows:

[0131] (1) A collection module 210, wherein the collection module 210 is used to collect first image information, wherein the first image information is used to represent the installation progress of the cable head.

[0132] (2) An acquisition module 220, wherein the acquisition module 220 is used to obtain pre-processed second image information based on the first image information.

[0133] (3) A first determination module 230, wherein the first determination module 230 is used to determine a first identification feature and a second identification feature based on the second image information, wherein the first identification feature is an object feature and the second identification feature is a gesture feature.

[0134] (4) A second determination module 240, which is used to determine the current cable head processing progress based on the first identification feature and the second identification feature.

[0135] Further, obtaining the preprocessed second image information based on the first image information includes sequentially performing damage cleaning, blur cleaning and similarity cleaning on the first image information to obtain the second image information.

[0136] Furthermore, the first determination module 230 is also used to determine the first recognition feature using the yolov8 model based on the second image information.

[0137] Furthermore, the first determination module 230 is also used to obtain a cable head installation and production training sample set, and to perform category labels, wherein the category labels are used to characterize dynamic gestures; the labeled training samples are used as input to construct a gesture recognition model, wherein the gesture recognition model includes a dynamic channel and a static channel, wherein the dynamic channel uses image processing technology and a CNNLSTM neural network to capture the motion trajectory of the gesture; the static channel uses a structural similarity index SSIM to determine the key frames of the gesture to construct a gesture recognition model combining the dynamic channel and the static channel; and the second recognition feature is determined based on the dynamic channel and the static channel.

[0138] The cable head processing progress monitoring device provided by the present disclosure includes an acquisition module, an acquisition module, a first determination module and a second determination module, wherein the acquisition module is used to acquire first image information, and the first image information is used to characterize the installation progress of the cable head; the acquisition module is used to obtain pre-processed second image information based on the first image information; the first determination module is used to determine a first recognition feature and a second recognition feature based on the second image information, wherein the first recognition feature is an object feature, and the second recognition feature is a gesture feature; the second determination module is used to determine the current cable head processing progress based on the first recognition feature and the second recognition feature; thereby, the cable head production process can be effectively identified, and the on-site supervision of the cable head can be promoted to be comprehensive, intelligent and standardized, which is beneficial to the management of the cable head production workflow and improves work efficiency.

[0139] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0140] To this end, a third embodiment of the present disclosure provides a storage medium, which is a computer-readable medium and stores a computer program. When the computer program is executed by a processor, the method provided by the embodiment of the present disclosure is implemented, including the following steps S11 to S14:

[0141] S11, collecting first image information, wherein the first image information is used to represent the installation progress of the cable head;

[0142] S12, obtaining preprocessed second image information based on the first image information;

[0143] S13, determining a first identification feature and a second identification feature based on the second image information, wherein the first identification feature is an object feature, and the second identification feature is a gesture feature;

[0144] S14 determines the current cable head processing progress based on the first identification feature and the second identification feature.

[0145] Furthermore, when the computer program is executed by a processor, other methods provided by any of the above embodiments of the present disclosure are implemented.

[0146] The cable head processing progress monitoring method provided in the present disclosure is applied, by collecting first image information, wherein the first image information is used to characterize the installation progress of the cable head; obtaining pre-processed second image information based on the first image information; and determining a first recognition feature and a second recognition feature based on the second image information, wherein the first recognition feature is an object feature, and the second recognition feature is a gesture feature; and then determining the current cable head processing progress based on the first recognition feature and the second recognition feature; it can effectively identify the cable head production process, promote the comprehensive, intelligent and standardized on-site supervision of the cable head, facilitate the cable head production workflow management, and improve work efficiency.

[0147] A fourth embodiment of the present disclosure provides an electronic device, such as Fig. 9 As shown, the electronic device at least includes a memory 310 and a processor 320. The memory 310 stores a computer program. The processor 320 implements the method provided by any embodiment of the present disclosure when executing the computer program on the memory 310. Exemplarily, the method executed by the electronic device computer program is as follows:

[0148] S21, collecting first image information, wherein the first image information is used to represent the installation progress of the cable head;

[0149] S22, obtaining preprocessed second image information based on the first image information;

[0150] S23, determining a first identification feature and a second identification feature based on the second image information, wherein the first identification feature is an object feature, and the second identification feature is a gesture feature;

[0151] S24 determines the current cable head processing progress based on the first identification feature and the second identification feature.

[0152] In specific implementation, the acquisition module 210, the acquisition module 220, the first determination module 230 and the second determination module 240 are stored in the memory as program units, and the processor executes the program units stored in the memory to implement corresponding functions.

[0153] The cable head processing progress monitoring method provided in the present disclosure is applied, by collecting first image information, wherein the first image information is used to characterize the installation progress of the cable head; obtaining pre-processed second image information based on the first image information; and determining a first recognition feature and a second recognition feature based on the second image information, wherein the first recognition feature is an object feature, and the second recognition feature is a gesture feature; and then determining the current cable head processing progress based on the first recognition feature and the second recognition feature; it can effectively identify the cable head production process, promote the comprehensive, intelligent and standardized on-site supervision of the cable head, facilitate the cable head production workflow management, and improve work efficiency.

[0154] The storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0155] The storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains at least two Internet Protocol addresses; sends a node evaluation request including at least two Internet Protocol addresses to a node evaluation device, wherein the node evaluation device selects an Internet Protocol address from the at least two Internet Protocol addresses and returns it; receives the Internet Protocol address returned by the node evaluation device; wherein the obtained Internet Protocol address indicates an edge node in a content distribution network.

[0156] Alternatively, the storage medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device: receives a node evaluation request including at least two Internet Protocol addresses; selects an Internet Protocol address from the at least two Internet Protocol addresses; and returns the selected Internet Protocol address; wherein the received Internet Protocol address indicates an edge node in a content distribution network.

[0157] Computer program code for performing operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the passenger computer, partially on the passenger computer, as a separate software package, partially on the passenger computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the passenger computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0158] It should be noted that the storage medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any storage medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0159] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0160] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0161] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0162] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0163] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0164] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0165] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0166] Multiple embodiments of the present disclosure are described in detail above, but the present disclosure is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications to the embodiments based on the concepts of the present disclosure, and these variations and modifications should all fall within the scope of protection required by the present disclosure.

Claims

1. A cable head processing progress monitoring method based on image recognition, characterized in that: The steps include: Collecting first image information, wherein the first image information is used to represent the installation progress of the cable head; Obtaining preprocessed second image information based on the first image information; Determine a first identification feature and a second identification feature based on the second image information, wherein the first identification feature is an object feature and the second identification feature is a gesture feature; The current cable head processing progress is determined based on the first identification feature and the second identification feature.

2. The cable head processing progress monitoring method based on image recognition according to claim 1 is characterized in that: Obtaining preprocessed second image information based on the first image information includes sequentially performing damage cleaning, blur cleaning and similarity cleaning on the first image information to obtain second image information.

3. The cable head processing progress monitoring method based on image recognition according to claim 1 is characterized in that: Determining the first identification feature based on the second image information includes: determining the first identification feature using the yolov8 model based on the second image information.

4. The cable head processing progress monitoring method based on image recognition according to claim 1 is characterized in that: Determining a second identification feature based on the second image information includes: Obtaining a cable head installation training sample set and performing category labeling, wherein the category label is used to characterize dynamic gestures; The labeled training samples are used as input to build a gesture recognition model, wherein the gesture recognition model includes a dynamic channel and a static channel. The dynamic channel uses image processing technology and CNNLSTM neural network to capture the motion trajectory of the gesture; the static channel uses the structural similarity index SSIM to determine the key frame of the gesture to build a gesture recognition model combining the dynamic channel and the static channel; A second identification feature is determined based on the dynamic channel and the static channel.

5. The cable head processing progress monitoring method based on image recognition according to claim 4 is characterized in that: Determining the second identification feature based on the dynamic channel and the static channel includes: Acquire dynamic gesture features based on dynamic channels; Obtain static gesture features based on static channels; A second recognition feature is determined based on the dynamic gesture feature and the static gesture feature.

6. The cable head processing progress monitoring method based on image recognition according to claim 5 is characterized in that: The acquiring of dynamic gesture features based on the dynamic channel includes: The two adjacent frames in the video sequence are differentiated to obtain the foreground image of the gesture motion area, the motion area graphic and the background image are differentiated to extract the motion target image, and the motion target image is compared with the threshold to obtain a binary image; A segmented gesture foreground video is obtained based on a binary image, and is input into a CNNLSTM network to extract motion features; the CNNLSTM architecture includes a convolution layer, a maximum pooling layer, a flattening layer, a long short-term memory LSTM layer, and a Softmax output layer, wherein the convolution layer and the maximum pooling layer are connected in parallel, and the input differential image sequence is flattened by the flattening layer and then input into the LSTM layer and the Softmax output layer in sequence, and the output of the Softmax output layer is used as the final output of the CNNLSTM architecture; The input differential image sequence first passes through the first convolutional layer to extract local features, then passes through the first maximum pooling layer to reduce the size of the feature map while retaining important features, and then passes through the second convolutional layer to further extract features; The output of the second maximum pooling layer passes through the flattening layer to flatten the feature map into a one-dimensional vector and input it into the LSTM layer. The LSTM layer uses past context information for processing to capture long-term dependencies in the time series. The output of the LSTM layer is fused with the features of the static channel output, and after passing through the Softmax activation function, the probability distribution of each gesture category is generated and the gesture label is assigned.

7. The cable head processing progress monitoring method based on image recognition according to claim 5 is characterized in that: The acquiring of static gesture features based on the static channel includes: The structural similarity index SSIM is used to calculate the similarity between adjacent frames, and the peak moment of the motion gesture is determined by analyzing the change trend of SSIM; After determining the gesture peak frame, the background in the gesture peak frame is removed using the background subtraction method; After removing the background, the gesture peak frame image is subjected to YCbCr color space analysis and threshold segmentation to extract the gesture; The position of the hand is located based on the extracted gesture.

8. The cable head processing progress monitoring method based on image recognition according to claim 5 is characterized in that: The second recognition feature is determined based on the dynamic gesture feature and the static gesture feature, including: splicing the output feature vectors extracted from the dynamic channel and the static channel, and inputting the spliced ​​feature vectors into a two-layer fully connected network, using the softmax function to normalize the output to obtain the probability of each gesture category, and selecting the category with the highest probability as the recognition result, wherein the softmax function is defined as follows: e is the base of natural logarithms, z i is the i-th element of the input vector z, p i is the i-th element of the output vector p, representing the probability of the i-th class.

9. The cable head processing progress monitoring method based on image recognition according to claim 1, characterized in that: Determining the current cable head processing progress based on the first identification feature and the second identification feature includes: comparing the first identification feature and the second identification feature, and if key object information appears in the image frame and a corresponding gesture is performed, the current step can be determined.

10. A cable head processing progress monitoring device based on image recognition, characterized in that: include: A collection module, used for collecting first image information, wherein the first image information is used for representing the installation progress of the cable head; An acquisition module, configured to acquire preprocessed second image information based on the first image information; A first determination module; configured to determine a first identification feature and a second identification feature based on the second image information, wherein the first identification feature is an object feature and the second identification feature is a gesture feature; The second determination module is used to determine the current cable head processing progress based on the first identification feature and the second identification feature.

Images