Deep learning-based drainage pipeline scene video key information extraction method and computer equipment

Abstract

The invention discloses a drainage pipeline scene video key information extraction method based on deep learning, and computer equipment, and the method comprises the steps: generating a text detection and recognition data set in a drainage pipeline scene, specifically adjusting the hyper-parameters of a text detection and recognition model, completing the training of a two-stage neural network, and obtaining the key information of the drainage pipeline scene video. The model fully learns knowledge in a drainage pipeline scene and can be well applied to the drainage pipeline scene; for a high-frame video, the number of sampling frames per second can be flexibly changed, a sampling frame number can be generated in advance, meanwhile, a multi-thread technology is adopted, the processing performance of an existing multi-core CPU is fully utilized, the processing speed of the high-frame video is greatly increased, and through an extraction strategy based on multi-frame data key information, the processing speed of the high-frame video is greatly improved. And the situation that blurred text information is extracted wrongly in the scene is avoided.

Description

Technical Field

[0001] The present invention relates to a method and computer equipment for extracting key information from drainage pipe scene videos based on deep learning, and in particular to utilizing optical character recognition technology to accurately and completely extract text information from complex scenes by integrating frame data per second of video. Background Art

[0002] Currently, most information is transmitted mainly through video streams. Therefore, the technology of extracting text information from videos has attracted the attention of many scholars. Optical character recognition is the basis for extracting key information from videos. With the continuous development of artificial intelligence in recent years, this field has also received widespread attention.

[0003] In order to solve the problem of extracting key information from videos under complex backgrounds, many solutions have been proposed by academic and industrial circles at home and abroad. Among them, the technical solutions that are closer to the present invention include: Diao Yuehua (Design and Implementation of Network Video Subtitle Extraction and Recognition System [D]. Beijing Jiaotong University, 2015) designed a subtitle positioning algorithm based on stroke width transformation, which improved text contrast through multi-frame fusion enhancement and Otsu (maximum inter-class difference method) binarization, and combined with a multi-level classifier to achieve text recognition. However, this method is susceptible to noise interference when processing low-resolution videos, and the computational complexity of feature matching is high, making it difficult to meet real-time requirements; Zhao Jie (Text Information Extraction Algorithm in Agricultural Science and Education Videos [J]. Northwest A&F University, 2018) proposed a text positioning method based on MSER (maximum stable extreme region), combined with projection segmentation and Tess The eract engine realizes text extraction in complex backgrounds, but this method relies on the preprocessing of multiple frames of images, and MSER (maximum stable extreme region) is sensitive to changes in illumination, and is prone to false detection in dynamic scenes; Zhang Shengmao (Text Information Extraction from Electronic Monitoring Videos of Offshore Fishing Vessels [J]. Fisheries Information and Strategy, 2020) samples actual scenes and uses jTessBoxEditor to generate training samples. Through the Tesseract engine, the font library is customized and the text is recognized, which realizes the extraction of time, location and other information in specific scenes. However, this solution relies on the fixed position of the text and extracts text information at a fixed position. Once the text information changes, errors will occur and the model cannot cope with dynamic scenes.

[0004] In summary, the existing video text detection and recognition solutions have the following shortcomings: (1) Most existing methods rely on traditional image processing technology, which is difficult to cope with the diversity of modern fonts, interference from complex backgrounds, and changes in text position information; (2) Some methods enhance recognition performance through multi-frame fusion, but the fusion strategy does not fully consider the temporal continuity of the video sequence, resulting in high computational delay and difficulty in meeting the needs of real-time applications; (3) Some methods obtain training samples by sampling in real scenes. This method relies on manual labor, is inefficient, and may cause mislabeling, which interferes with subsequent model training. (4) Most text recognition technologies rely on the Tesseract open source optical character recognition engine. The recognition technologies currently used by Tesseract are LSTM (Long Short-Term Memory Network) and CNN (Convolutional Neural Network), which extract the semantic features of characters through end-to-end learning. However, CNN (Convolutional Neural Network) extracts local features through convolution kernels, but has weak modeling capabilities for the global structure of characters (such as long-distance stroke associations and overall layout). LSTM has high time complexity when processing long texts ( O ( n ), n is the sequence length), which is difficult to meet the needs of real-time applications.

[0005] Video text detection and recognition is a fundamental method for extracting text information from videos, and has broad practical application across various industries. However, current methods are slow for processing high-frame-rate videos and lack targeted solutions for complex scenarios such as insufficient lighting, background interference, and blurred text. This application addresses this issue by proposing a solution. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings and deficiencies of the existing technology and provide a method and system for extracting key information from drainage pipe scene videos based on deep learning. This method can effectively handle the frame extraction strategy of high-frame-rate videos and the detection and recognition strategy of multiple frames in complex scenes.

[0007] The first aspect of the present invention is to provide a method for extracting key information from a drainage pipe scene video based on deep learning, the method comprising: Collect images of text-free areas in the drainage pipe scene, augment them using an augmentation strategy, and obtain a background image set. Establish text size set, font set, and character color change operation actions in the drainage pipe scene; Collect corpus and generate text dictionary to get dictionary Dict and a corpus collection, the corpus collection including the following elements: a task address text collection, a pipeline number text collection, a pipeline material text collection, a distance and speed text collection, and a date and time text collection; Create text detection datasets and text recognition datasets based on text size sets, font sets, character color transformation operation sets, and corpus sets; Train the text detection model, based on the differentiable binarization text detection model, train the text detection dataset to obtain the optimal text detection model; Train the text recognition model, and train the text recognition dataset based on the single-vision text recognition model to obtain the optimal text recognition model; Multi-frame image video information extraction, for a given drainage pipe video V， Interval sampling is performed, and the sampling process is multi-threaded and parallelized in seconds. That is, each thread is responsible for processing the frame sampling task within a certain second in the video, and the entire video frame set is obtained through multi-threaded parallel sampling; Perform frame-by-frame detection and recognition operations on all sampled video frame sets, take out one frame image from all video frame sets in turn and delete the frame from the set, use the optimal detection model for reasoning to obtain a text frame set for one frame image, crop the position of each text line according to the text frame and save it to obtain a text line image set; then use the optimal recognition model to perform frame-by-frame reasoning on the text line image set to obtain the text and confidence set. The comprehensive detection and recognition results are set; repeat this step until all the video frame sets are empty, and the detection and recognition result set for all seconds of the video is obtained; The recognition results in each second of the detection and recognition result set of all seconds of the video are aggregated into similar frames to obtain a similar frame set, and the best result is selected from the similar frame set.

[0008] It is further configured that the augmentation strategy includes random horizontal flipping, random rotation, random scaling, random cropping, random color space transformation, and random addition of Gaussian noise of varying degrees.

[0009] It is further provided that the method for generating the text detection dataset and the text recognition dataset includes: Step 1.4.1: Randomly select one text message from each of the five types of text elements in the corpus to form a text collection texts , randomly extract a background image from the background image set, randomly select a font from the font set, randomly select a font size from the text size set, specify the position of each type of text box, select a single character color according to the color change operation of the character, and then write the text character by character into the various text boxes in the background image, resulting in five types of text boxes of box Text Box Collection boxes , fusion pictures pic:; Step 1.4.2 ： According to the text box position information, the fused image pic The text box area The domains are cropped separately to obtain the image patch and corresponding text information text ; Step 1.4.3: Repeat step 1.4.1 to obtain the text detection dataset; repeat steps 1.4.1 and 1.4.2 to obtain the text recognition dataset.

[0010] It is further provided that the optimal text detection model is a training method comprising: using a text detection model based on differentiable binarization, using the text detection dataset as training data, and giving hyperparameters in advance thresh , hyperparameters box_thresh , learning rate learning_rate det , batch size n dbatch and the number of training rounds epoch det , and obtain it through deep learning training iterations of forward propagation, error calculation, and reverse update.

[0011] The further configuration is that the training method of the optimal text recognition model includes: using a single visual text recognition model, using the text recognition dataset as training data, and giving hyperparameters in advance max_text_length , learning rate learning_ rate rec , batch size n rbatch and the number of training rounds epoch rec , and obtain it through deep learning training iterations of forward propagation, error calculation, and reverse update.

[0012] The present invention also provides a computer device, which includes one or more processors and a memory storing computer program instructions, and when the computer program instructions are executed, the processor performs the steps of the above-mentioned method for extracting key information from drainage pipe scene videos based on deep learning.

[0013] By implementing the above scheme, the present invention provides a method for extracting key information from drainage pipe scene videos based on deep learning. It generates a text detection and recognition data set in the drainage pipe scene, and specifically adjusts the hyperparameters of the text detection and recognition model to complete two-stage neural network training, so that the model can fully learn the knowledge in the drainage pipe scene and can be better applied in the drainage pipe scene. For high-frame videos, the number of sampling frames per second can be flexibly changed, and the sampling frame number is generated in advance. At the same time, multi-threading technology is adopted to make full use of the processing performance of current multi-core CPUs, greatly accelerating the processing speed of high-frame videos. Through an extraction strategy based on key information of multi-frame data, the error of extracting unclear text information in the scene is avoided.

[0014] In addition, this method extracts multi-frame image video information through the "sampling-detection and recognition-multi-frame aggregation-optimal screening" process, combined with IoU threshold clustering and confidence / text length dual screening to solve problems such as blurred text and position changes in drainage pipe videos, ensuring the accuracy and robustness of the extraction results.

[0015] The present invention will be further described below with reference to the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is an overall flow chart of an embodiment of the present invention; Figure 2 This is a sample image of a video frame in a drainage pipe scene; Figure 3 This is a sample image of the detection dataset obtained in step 1.4; Figure 4 This is a sample image of the recognition dataset obtained in step 1.4; Figure 5 for Figure 2 The threshold map obtained in step 2.1.1; Figure 6 for Figure 3 The binary feature map obtained in step 2.1.1; Figure 7 Generate a flowchart for the drainage pipe scene text detection and recognition dataset; Figure 8 Flowchart for text detection and recognition model training; Figure 9 The flowchart of video information extraction based on multi-frame images. DETAILED DESCRIPTION

[0017] The present invention is specifically described below through examples, which are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical engineers in this field can make some non-essential improvements and adjustments to the present invention based on the contents of the above invention.

[0018] The following is a detailed description of the method for extracting key information from a drainage pipe scene video using an example. The steps are as follows: 1. Drainage pipe scene text detection and recognition dataset generation, which mainly includes the following steps: Step 1.1: Collect a batch of text-free area images in the drainage pipe scene, and augment them using a pre-given augmentation strategy. The number of images is n bg Background image collection BgIn this example, the augmentation strategies include random horizontal flipping, random rotation, random scaling, random cropping, random color space transformation, and random addition of different degrees of Gaussian noise. n bg =3000; Step 1.2: The text size set in the drainage pipe scene is fontSize , font collection is font , character color change operation f In this example, the text size set fontSize ∈[22, 30], font set font Includes the following formats: (1) Chinese: Songti, Ipix; (2) English: Arial, Perfect DOS VGA 437; (3) Chinese and English: Adelle Sans CHI Lt; Step 1.3: Collect corpus and generate text dictionary to get dictionary Dict and corpus collection Yl= { text dz , text bh , text cz , text jl , text sj}, text dz 、 text bh 、 text cz 、 text jl 、 text sj They represent the task address text set, pipeline number text set, pipeline material text set, distance and speed text set, and date and time text set respectively; Step 1.4: Synthetic Text Detection Dataset det and recognition datasets Dataset rec , according to the background image set Bg, Text size collection fontSize , font collection is font , special transformation operations f and corpus information Yl To create a text detection dataset and a recognition dataset, the specific steps are as follows: Step 1.4.1: From YlA text message is randomly selected from each of the five categories of text in the collection to form a text collection texts , randomly from Bg A background image is randomly drawn from font Select fonts randomly from fontSize Select font size fs , specify the position of various text boxes box , according to the operation f Select a single character color, and then write the text character by character into various text boxes in the background image to get a text box collection boxes , fusion pictures pic ,gather boxes Contains five types of text boxes box , box The values are as follows: box =(( pos X , pos Y ), ( pos X + len , pos Y ), ( pos X + [[ID=�6]]len , pos Y + fs ), ( pos X , pos Y + fs ) ( pos X , pos Y ) is the anchor point of the upper left corner of the text box, ( pos X , pos Y ) varies according to the text type, as shown below: (1) Task address class: ( pos X , pos Y )=( ac , ac ) (2) Pipeline number category: ( pos X , pos Y )=( ac , ac + fs ) (3) Pipe material:pos X , pos Y )=( ac , ac +2× fs ) (4) Distance and speed: pos X , pos Y )=( ac + offset 1, ac +2× fs + offset 2) (5) Date and time class: ( pos X , pos Y ) = ( ac + offset 1+ offset 3, ac +2 fs + offset 2 + offset 4) in, ac is a fixed value given in advance, offset 1. offset 2. offset 3. offset 4 is a random offset within a given range. In this example, ac =20, offset 1. offset 3∈[300, 500], offset 2 , offset 4 ∈[100, 200]; len is the text length, calculated by formula (1): in n char is the length of the text; select the color of a single character as shown in formula (2): in, thresh is a pre-given character color judgment threshold. In this example, thresh =0.55, brightness is the average brightness of the background area of a single character, calculated by formula (3): r i is the pixel ratio of each grayscale value, i ∈[0, 255]: P yes H The sum of the number of pixels corresponding to all grayscale values in : H It is the histogram of the grayscale image of the background area where a single character is located. H [ i ] indicates that the gray value is i The number of pixels; Step 1.4.2: Based on the text box position information, merge the image pic The text box areas in the image are cropped separately to obtain the image. patch and corresponding text information text ; Step 1.4.3: Repeat step 1.4.1 n det Second text detection dataset Dataset det ={( pic , texts , boxes ) j | j =1,2,…, n det}, n det To test the number of samples in the dataset, repeat steps 1.4.1 and 1.4.2. n rec Second text recognition dataset Dataset rec ={( patch , text ) k | k =1,2,…, n rec}, n rec To identify the number of samples in the dataset, in this example n det =5000, n rec =500,000; 2. Text detection and recognition model training mainly includes the following steps: Step 2.1: Training the text detection model. The present invention selects a text detection model based on Differentiable Binarization. The training data set is Dataset det , given hyperparameters in advance thresh , hyperparameters box_thresh , learning ratelearning_rate det , batch size n dbatch and the number of training rounds epoch det , in this example thresh =0.15, box_thresh =0.5, learning_rate det =0.0005, n dbatch =8, epoch det =500, the training process includes the following steps: Step 2.1.1: From the dataset Dataset det Take out an image f ∈ R C×H×W , input into the Differentiable Binarization detection model for forward propagation to obtain the text threshold map T mn , text probability map P mn , for text threshold map T mn , text probability map P mn Calculate the text binarization graph using the differentiable binarization method B mn , m , n ∈R H ×W ; Step 2.1.2: Binarize the text image B mn Use Vatti clipping algorithm to expand and get text box information; Step 2.1.3: Calculate the forward propagation error based on the loss function of the Differentiable Binarization text detection model, and then perform backpropagation based on the error. Step 2.1.4: Repeat steps 2.1.1 to 2.1.3 until completed iter det Iterations: Get the optimal detection model Model det ; Step 2.2: Training the text recognition model. The present invention selects a method based on SVTR (single vision text recognition model) and the training data set is Dataset rec, the text dictionary is Dict , given hyperparameters in advance max_text_length , learning rate learning_rate rec , batch size n rbatch and the number of training rounds epoch rec , in this example max_text_length =50, n rbatch =128, learning_rate rec =0.0001, epoch rec =500, the training steps are as follows: Step 2.2.1: From the dataset Dataset rec Take out an image f ∈ R C×H×W , input into the SVTR text recognition model for forward propagation to obtain the predicted character sequence CC ; Step 2.2.2: Calculate the forward propagation error value based on the loss function of the SVTR text recognition model, and then perform backpropagation based on the error; Step 2.2.3: Repeat steps 2.2.1 to 2.2.2 until completed iter rec Iterations: Get the optimal recognition model Model rec ; 3. Extracting video information based on multiple frames of images mainly includes the following steps: Step 3.1: For a given drainage pipeline video V , the video frame rate is fps , first of all, sampling is performed, and the sampling strategy is every interval val The frame sampling is done in parallel by multi-threads in seconds, that is, each thread is responsible for processing the frame sampling task within a certain second in the video, and the entire video frame set is obtained through multi-threaded parallel sampling. L frame , in this example val =3: n second Indicates the total number of seconds of the video. x j For the j A collection of frame numbers per second: No. j Start frame of second : The number of frames sampled per second n frame : Step 3.2: For the sampled video frame set L frame ={ x j | j =1,2,…, n second}, perform frame-by-frame detection and recognition operations, the processing steps are as follows: Step 3.2.1: Take out the collection one by one L frame A frame of image and delete the frame from the set, using the optimal detection model Model det Reasoning can get a set of text boxes for a frame of image Boxes ={ Box p | p =1,2,3,…, n patch}, n patch Indicates the number of text boxes in a frame image. Each text line is cropped and saved according to the text box to obtain a text image collection. Imgs ={ img p | p =1,2,3,…, n patch}; Step 3.2.2: For the text line image collection Imgs , using the optimal recognition model Model rec Perform image-by-image reasoning to identify the text and confidence set TC ={( Text p , Con p )| p =1,2,3,…, n patch}, Text p and Con p Indicates the p The text line image is output by the recognition model and the confidence level, and the comprehensive detection and recognition results are obtained as a set BTC ={( Box p , Textp , Con p )| p =1,2,3,…, n patch}; Repeat steps 3.2.1 and 3.2.2 for each consecutive treatment. n frame After the frame image is captured, the detection and recognition of the video content of a certain second is completed, and then the detection and recognition result set within that second is obtained. BTCS ={( Box pq , Text pq , Con pq )| p =1,2,3,…, n patch , q = 1,2,…, n frame}, where the triple ( Box pq , Text pq , Con pq ) indicates the first q Frame p The position of each text box, its text content and its confidence; until L frame If the set is empty, get the detection and recognition result set of all seconds of the video DR ={ BTCS j | j =1,2,…, n second}; Step 3.3: Detection and recognition result set for all seconds of the video DR , it is necessary to further extract the optimal recognition results within each second. The processing steps are as follows: Step 3.3.1: Take out the collection DR A subset BTCS , perform aggregation of similar boxes, specifically creating a new empty collection SP and from the collection BTCS Extract the text box collection of the first frame Box 1 ={ Box p1 | p =1,2,3,…, n patch}, initialize the collection accordingly SP , andBox 1 from BTCS Removed, SP Divided into n patch Subsets: SP BTCS ={{ Box 11},{ Box 21},…, { Box p1}}, then from BTCS Get the text box collection of the subsequent frame and remove it from BTCS Delete, for each text box in the collection, in turn with SP The first text box of each sub-collection is judged as the same type. If it belongs to the same type, it is added to the corresponding sub-collection. If it does not belong to the same type as the first box of all existing sub-collections, it is added to the corresponding sub-collection. SP Create a new subcollection in the collection and add the text box, repeat this step until the collection BTCS Empty, get SP ={ sp o ( Text , Con )| o =1,2,3,…, n kind},in sp o Indicates the o A collection of similar frames, n kind express SP the number of neutron sets; To determine whether the boxes are of the same type, use formulas (13), (14), (15), and (16): Area ( Box ) represents the area of the text box, Inter ( Box , Box ) represents the area of the intersection of two text boxes, Union ( Box , Box ) represents the area of the union of two text boxes, ∩ represents the intersection operation of two text boxes, and ∪ represents the union operation of two text boxes. thresh Box Indicates the threshold for determining whether it is a similar box. In this example thresh Box =0.8; Step 3.3.2: SP Select the optimal result, and the selection rule is shown in formula (17): Indicates the o The best text information result in the same frame, MAX L (·) means to get the longest text information set in the collection, MAX C (·) means obtaining the text information set with the highest confidence in the set; Repeat steps 3.3.1 and 3.3.2 until the DR If it is empty, the video information extraction result set is obtained. Result ={ | o =1,2,3,…, n kind , j =1,2,…, n second}, Indicates the j Seconds o The best result among the same kind of boxes.

Claims

1. A deep learning-based method for extracting key information from drainage pipe scene videos, characterized by The method comprises: Collect images of text-free areas in the drainage pipe scene, augment them using an augmentation strategy, and obtain a background image set. Establish text size set, font set, and character color change operations in the drainage pipe scene; Collect corpus and generate text dictionary to get dictionary Dict and a corpus collection, the corpus collection including the following elements: a task address text collection, a pipeline number text collection, a pipeline material text collection, a distance and speed text collection, and a date and time text collection; Create text detection datasets and text recognition datasets based on text size sets, font sets, character color transformation operation sets, and corpus sets; Train the text detection model, based on the differentiable binarization text detection model, train the text detection dataset to obtain the optimal text detection model; Train the text recognition model, and train the text recognition dataset based on the single-vision text recognition model to obtain the optimal text recognition model; Multi-frame image video information extraction, for a given drainage pipe video V , perform interval sampling, and the sampling process is multi-threaded parallel processing in seconds, that is, each thread is responsible for processing the frame sampling task within a certain second in the video, and the entire video frame set is obtained through multi-threaded parallel sampling; Perform frame-by-frame detection and recognition operations on all sampled video frame sets, take out one frame image from all video frame sets in turn and delete the frame from the set, use the optimal detection model for reasoning to obtain a text frame set for one frame image, crop the position of each text line according to the text frame and save it to obtain a text line image set; then use the optimal recognition model to perform frame-by-frame reasoning on the text line image set to obtain the text and confidence set. The comprehensive detection and recognition results are set; repeat this step until all the video frame sets are empty, and the detection and recognition result set for all seconds of the video is obtained; The recognition results in each second of the detection and recognition result set of all seconds of the video are aggregated into similar frames to obtain a similar frame set, and the best result is selected from the similar frame set.

2. The method for extracting key information from drainage pipe scene videos based on deep learning according to claim 1 is characterized by: The augmentation strategies include random horizontal flipping, random rotation, random scaling, random cropping, random color space transformation, and random addition of different degrees of Gaussian noise.

3. The method for extracting key information from drainage pipe scene videos based on deep learning according to claim 1 is characterized by: The method for generating the text detection dataset and the text recognition dataset includes: Step 1.4.1: Randomly select one text message from each of the five types of text elements in the corpus to form a text collection texts , randomly extract a background image from the background image set, randomly select a font from the font set, randomly select a font size from the text size set, specify the position of each type of text box, select a single character color according to the color change operation of the character, and then write the text character by character into the various text boxes in the background image, resulting in five types of text boxes box Text Box Collection boxes , fusion pictures pic ; Step 1.4.2: According to the text box position information, the fused image pic The text box area The domains are cropped separately to obtain the image patch and corresponding text information text ； Step 1.4.3: Repeat step 1.4.1 to obtain the text detection dataset; repeat steps 1.4.1 and 1.4.2 to obtain the text recognition dataset.

4. The method for extracting key information from drainage pipe scene videos based on deep learning according to claim 1 is characterized by: The optimal text detection model is a training method comprising: using a text detection model based on differentiable binarization, using the text detection dataset as training data, and giving hyperparameters in advance thresh , hyperparameters box_thresh , learning rate learning_rate det , batch size n dbatch and the number of training rounds epoch det , and obtain it through deep learning training iterations of forward propagation, error calculation, and reverse update.

5. The method for extracting key information from drainage pipe scene videos based on deep learning according to claim 4 is characterized by: The training method of the optimal text recognition model includes: using a single visual text recognition model, using the text recognition dataset as training data, and giving hyperparameters in advance max_text_length , learning rate learning_rate rec , batch size n rbatch and the number of training rounds epoch rec , and obtain it through deep learning training iterations of forward propagation, error calculation, and reverse update.

6. A computer device, characterized in that: The computer device includes one or more processors and a memory storing computer program instructions, and when the computer program instructions are executed, the processor performs the steps of the method for extracting key information from drainage pipe scene videos based on deep learning as claimed in any one of claims 1 to 5.

Images