Dynamic semantic segmentation model training method, semantic segmentation method and device

Abstract

The invention provides a training method of a dynamic semantic segmentation model and a semantic segmentation method and device of a coal mine fully mechanized coal mining face, and the method comprises the steps: carrying out the preprocessing of historical video data of the coal mine fully mechanized coal mining face, and obtaining a video sequence; performing optical flow calculation processing on the video sequence to obtain training feature data; obtaining a dynamic semantic segmentation model according to the first type of training data and the second type of training data of the training feature data and the original model; in the pre-training stage, model training is carried out based on a first type of training data to obtain an intermediate dynamic semantic segmentation model; and in the label labeling stage, semantic segmentation label labeling is performed on the second type of training data based on the intermediate dynamic semantic segmentation model, and the first type of training data is updated. The device is used for executing the method. According to the training method of the dynamic semantic segmentation model and the semantic segmentation method and device of the fully mechanized coal mining face, the accuracy and generalization ability of dynamic semantic segmentation of the fully mechanized coal mining face are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a training method for a dynamic semantic segmentation model, and a semantic segmentation method and device for a fully mechanized coal mining face. Background Art

[0002] In coal mine comprehensive mining operations, accurate identification of semantic information in the scene is of great significance to the realization of intelligent coal mines and is crucial to ensuring safe production.

[0003] Most traditional semantic segmentation methods are based on single-frame image processing, and the accuracy of semantic segmentation is low. In the prior art, the encoder-decoder architecture of the DeepLabV3 algorithm is used to capture the contextual information of the image through multi-scale and multi-level feature fusion. The encoder part usually uses a pre-trained deep convolutional neural network to extract the features of the input image. The decoder part is responsible for mapping these features back to the resolution of the original image to achieve pixel-level segmentation. Since the video stream is divided into independent image frames, it is equivalent to processing still pictures, and the accuracy of semantic segmentation is not high. Therefore, how to train a semantic segmentation model that can be applied to the comprehensive mining face for dynamic semantic segmentation has become an important issue that needs to be solved in this field. Summary of the invention

[0004] In response to the problems in the prior art, the embodiments of the present invention provide a training method for a dynamic semantic segmentation model, a semantic segmentation method and device for a coal mine fully mechanized mining face, which can at least partially solve the problems existing in the prior art.

[0005] In a first aspect, the present invention proposes a training method for a dynamic semantic segmentation model, which is applied to a fully mechanized mining face in a coal mine, comprising:

[0006] Preprocessing historical video data of a fully-mechanized coal mining face to obtain a video sequence of the fully-mechanized coal mining face;

[0007] Performing optical flow calculation processing on the video sequence of the coal mine fully mechanized mining working face to obtain training feature data of the coal mine fully mechanized mining working face; the training feature data includes a first type of training data and a second type of training data, the first type of training data has a corresponding semantic segmentation label, and the second type of training data lacks a corresponding semantic segmentation label; wherein the semantic segmentation label is set based on the coal mine fully mechanized mining working face;

[0008] A dynamic semantic segmentation model is obtained based on the first category of training data, the second category of training data and the original model; wherein, obtaining the dynamic semantic segmentation model based on the first category of training data, the second category of training data and the original model includes multiple trainings, each training includes a pre-training stage and a labeling stage; the pre-training stage performs model training based on the first category of training data to obtain an intermediate dynamic semantic segmentation model; the labeling stage performs semantic segmentation labeling on the second category of training data based on the intermediate dynamic semantic segmentation model, and screens out the second category of training data with valid semantic segmentation labels to update the first category of training data.

[0009] In a second aspect, the present invention proposes a semantic segmentation method for a fully mechanized coal mining face, which adopts the dynamic semantic segmentation model described in any of the above embodiments, and includes:

[0010] Based on the video data of the fully mechanized mining face of the coal mine, the current frame video data is obtained;

[0011] Preprocessing the current frame video data to obtain a current frame image corresponding to the current frame video data;

[0012] Performing optical flow calculation processing on a current frame image corresponding to the current frame video data and a previous frame image of the current frame image corresponding to the current frame video data to obtain predicted feature data corresponding to the current frame video data;

[0013] According to the predicted feature data corresponding to the current frame video data and the dynamic semantic segmentation model, a semantic segmentation result corresponding to the current frame video data is obtained.

[0014] In a third aspect, the present invention provides a training device for a dynamic semantic segmentation model, which is applied to a fully mechanized mining face in a coal mine, comprising:

[0015] A preprocessing unit, used for preprocessing historical video data of a fully-mechanized coal mining working face to obtain a video sequence of the fully-mechanized coal mining working face;

[0016] An optical flow calculation processing unit is used to perform optical flow calculation processing on the video sequence of the coal mine fully mechanized mining working face to obtain training feature data of the coal mine fully mechanized mining working face; the training feature data includes a first type of training data and a second type of training data, the first type of training data has a corresponding semantic segmentation label, and the second type of training data lacks a corresponding semantic segmentation label; wherein the semantic segmentation label is set based on the coal mine fully mechanized mining working face;

[0017] A training unit is used to obtain a dynamic semantic segmentation model based on the first category of training data, the second category of training data and the original model; wherein, obtaining the dynamic semantic segmentation model based on the first category of training data, the second category of training data and the original model includes multiple trainings, and each training includes a pre-training stage and a labeling stage; the pre-training stage performs model training based on the first category of training data to obtain an intermediate dynamic semantic segmentation model; the labeling stage performs semantic segmentation labeling on the second category of training data based on the intermediate dynamic semantic segmentation model, and screens out the second category of training data with valid semantic segmentation labels to update the first category of training data.

[0018] In a fourth aspect, the present invention provides a semantic segmentation device for a fully mechanized coal mining face, comprising:

[0019] The first acquisition unit is used to obtain video data based on the fully mechanized mining working face of the coal mine and obtain current frame video data;

[0020] A second obtaining unit, configured to preprocess the current frame video data to obtain a current frame image corresponding to the current frame video data;

[0021] A third obtaining unit is used to perform optical flow calculation processing on a current frame image corresponding to the current frame video data and a previous frame image of the current frame image corresponding to the current frame video data to obtain prediction feature data corresponding to the current frame video data;

[0022] The semantic segmentation unit is used to obtain the semantic segmentation result corresponding to the current frame video data according to the predicted feature data corresponding to the current frame video data and the dynamic semantic segmentation model.

[0023] In a fifth aspect, the present invention provides a computer device comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the program to implement the training method of the dynamic semantic segmentation model described in any of the above embodiments, or the semantic segmentation method of the coal mine comprehensive mining working face.

[0024] In a sixth aspect, the present invention provides a computer-readable storage medium, which stores a computer program / instruction. When the computer program / instruction is executed by a processor, it implements the training method of the dynamic semantic segmentation model described in any of the above embodiments, or the semantic segmentation method of a coal mine comprehensive mining working face.

[0025] In the seventh aspect, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the training method of the dynamic semantic segmentation model described in any of the above embodiments, or the semantic segmentation method of a coal mine comprehensive mining working face.

[0026] The training method of the dynamic semantic segmentation model and the semantic segmentation method and device of the coal mine fully mechanized mining working face provided in the embodiment of the present invention pre-process the historical video data of the coal mine fully mechanized mining working face to obtain the video sequence of the coal mine fully mechanized mining working face; perform optical flow calculation processing on the video sequence of the coal mine fully mechanized mining working face to obtain the training feature data of the coal mine fully mechanized mining working face; the training feature data includes a first type of training data and a second type of training data, the first type of training data has a corresponding semantic segmentation label, and the second type of training data lacks a corresponding semantic segmentation label; wherein the semantic segmentation label is set based on the coal mine fully mechanized mining working face; according to the first type of training data, the second type of training data and The original model is used to obtain a dynamic semantic segmentation model; wherein, the dynamic semantic segmentation model is obtained according to the first type of training data, the second type of training data and the original model, and includes multiple trainings, each training includes a pre-training stage and a labeling stage; the pre-training stage is to perform model training based on the first type of training data to obtain an intermediate dynamic semantic segmentation model; the labeling stage is to perform semantic segmentation labeling on the second type of training data based on the intermediate dynamic semantic segmentation model, and screen out the second type of training data with valid semantic segmentation labels to update the first type of training data, which can make full use of video sequences and optical flow information, and improve the accuracy and generalization ability of dynamic semantic segmentation of comprehensive mining working faces. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 It is a flowchart of the training method of the dynamic semantic segmentation model provided by the first embodiment of the present invention.

[0029] Figure 2 It is a flowchart of a training method for a dynamic semantic segmentation model provided in the second embodiment of the present invention.

[0030] Figure 3 It is a flowchart of a training method for a dynamic semantic segmentation model provided in the third embodiment of the present invention.

[0031] Figure 4 It is a flowchart of a training method for a dynamic semantic segmentation model provided in the fourth embodiment of the present invention.

[0032] Figure 5 It is a flowchart of a training method for a dynamic semantic segmentation model provided in the fifth embodiment of the present invention.

[0033] Figure 6 It is a flowchart of a training method for a dynamic semantic segmentation model provided in the sixth embodiment of the present invention.

[0034] Figure 7 It is a flowchart of obtaining points deduction corresponding to training feature data provided by the seventh embodiment of the present invention.

[0035] Figure 8 It is a flowchart of a training method for a dynamic semantic segmentation model provided in the eighth embodiment of the present invention.

[0036] Fig. 9 It is a flowchart of a training method for a dynamic semantic segmentation model provided in the ninth embodiment of the present invention.

[0037] Fig.10 It is a flow chart of the semantic segmentation method of a coal mine fully mechanized mining face provided in the tenth embodiment of the present invention.

[0038] Fig.11 It is a structural diagram of a training device for a dynamic semantic segmentation model provided in the eleventh embodiment of the present invention.

[0039] Fig.12 It is a structural diagram of a training device for a dynamic semantic segmentation model provided in the twelfth embodiment of the present invention.

[0040] Fig.13 It is a structural diagram of a training device for a dynamic semantic segmentation model provided in the thirteenth embodiment of the present invention.

[0041] Fig.14 It is a structural diagram of a training device for a dynamic semantic segmentation model provided in the fourteenth embodiment of the present invention.

[0042] Fig.15 It is a structural diagram of a training device for a dynamic semantic segmentation model provided in the fifteenth embodiment of the present invention.

[0043] Fig.16 It is a structural schematic diagram of a semantic segmentation device for a coal mine fully mechanized mining face provided in the sixteenth embodiment of the present invention.

[0044] Fig.17 It is a schematic diagram of the physical structure of an electronic device provided by the seventeenth embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined arbitrarily with each other. The acquisition, storage, use, and processing of data in the technical scheme in this application all comply with the relevant provisions of laws and regulations.

[0046] In order to facilitate understanding of the technical solution provided by this application, the relevant contents of the technical solution of this application are first explained below.

[0047] In traditional semantic segmentation, analysis usually relies on static single-frame images, which inevitably ignores the key clues carried by the motion information of adjacent frames. For example, for the visual system of organisms, a continuous video stream can capture richer context and dynamic changes than isolated images, thereby enabling more accurate target recognition. To this end, this application proposes a dynamic semantic segmentation method based on video sequences, which effectively enhances the accuracy and reliability of the recognition results.

[0048] In order to introduce motion information in the video, the present application performs optical flow processing on the video frames, and uses the results of the optical flow algorithm as training data. Since a large number of labeled samples are required for training the model, a large number of label annotations are manually performed, which is costly and inefficient. In order to improve the efficiency of label annotation and reduce costs, the present invention uses a small amount of labeled training data to train the model, and then uses the model to perform semantic segmentation label prediction on the unlabeled training data, evaluate the data with valid semantic segmentation prediction results, update the labeled training data, repeat the above process many times, and finally train a dynamic semantic segmentation model. The technical solution provided by the present invention can make full use of video sequences and optical flow information to improve the accuracy and generalization ability of dynamic semantic segmentation of comprehensive mining working faces. At the same time, the training method of semi-supervised learning can reduce the dependence on data annotation and improve the practicality of the network model. The present application has important practical application value for coal mine safety production.

[0049] The specific implementation process of the training method of the dynamic semantic segmentation model provided by the embodiment of the present invention is described below with the server as the execution subject. It is understandable that the execution subject of the training method of the dynamic semantic segmentation model provided by the embodiment of the present invention is not limited to the server.

[0050] Figure 1 is a flow chart of a training method for a dynamic semantic segmentation model provided by the first embodiment of the present invention, such as Figure 1 As shown, the training method of the dynamic semantic segmentation model provided by the embodiment of the present invention includes:

[0051] S101, preprocessing historical video data of a fully-mechanized mining face of a coal mine to obtain a video sequence of the fully-mechanized mining face of the coal mine;

[0052] Specifically, historical video data of a coal mine fully-mechanized mining face can be collected, and the server can pre-process the historical video data of the coal mine fully-mechanized mining face to obtain a video sequence of the coal mine fully-mechanized mining face, where the video sequence includes a series of continuous images. The pre-processing includes but is not limited to format conversion, normalization processing, etc., which are set according to actual needs and are not limited in the embodiments of the present invention.

[0053] S102, performing optical flow calculation processing on the video sequence of the coal mine fully mechanized mining working face to obtain training feature data of the coal mine fully mechanized mining working face; the training feature data includes a first type of training data and a second type of training data, the first type of training data has a corresponding semantic segmentation label, and the second type of training data lacks a corresponding semantic segmentation label; wherein the semantic segmentation label is set based on the coal mine fully mechanized mining working face;

[0054] Specifically, the server performs optical flow calculation on the video sequence of the coal mine fully-mechanized mining working face, obtains the motion information between each frame image and the previous frame image in the video sequence of the coal mine fully-mechanized mining working face, and uses the motion information between each frame image and the previous frame image and each frame image as a training feature data, and the motion information between each frame image and the previous frame image and each frame image constitute the training feature data of the coal mine fully-mechanized mining working face. The training feature data of the coal mine fully-mechanized mining working face includes a first type of training data and a second type of training data. The first type of training data is manually annotated and has corresponding semantic segmentation labels, and the second type of training data lacks corresponding semantic segmentation labels. The semantic segmentation labels are set based on the coal mine fully-mechanized mining working face, and may include categories such as support guard plates, scraper conveyors, rollers, coal, trough sides, rails, cable troughs, etc., and are set according to actual needs, and the embodiments of the present invention do not limit them.

[0055] For example, the first type of training data accounts for 5% of the training feature data of the coal mine fully mechanized mining working face, and the remaining training feature data of the coal mine fully mechanized mining working face is used as the second type of training data.

[0056] S103. Obtain a dynamic semantic segmentation model based on the first category of training data, the second category of training data and the original model; wherein, obtaining the dynamic semantic segmentation model based on the first category of training data, the second category of training data and the original model includes multiple trainings, and each training includes a pre-training stage and a labeling stage; the pre-training stage performs model training based on the first category of training data to obtain an intermediate dynamic semantic segmentation model; the labeling stage performs semantic segmentation labeling on the second category of training data based on the intermediate dynamic semantic segmentation model, and screens out the second category of training data with valid semantic segmentation labels to update the first category of training data.

[0057] Specifically, since the second category of training data does not have corresponding semantic segmentation labels and cannot be directly used for model training, in the pre-training stage of the first training, the original model is first trained with the first category of training data with semantic segmentation labels to obtain an intermediate dynamic semantic segmentation model. In the labeling stage of the first training, the second category of training data is labeled with the intermediate dynamic semantic segmentation model obtained in the pre-training stage to screen out the second category of training data with valid semantic segmentation labels. The first category of training data is updated based on the second category of training data with valid semantic segmentation labels, thereby increasing the number of training feature data included in the first category of training data. Among them, the original model can adopt the DeeplabV3+ model, which is selected according to actual needs and is not limited in the embodiments of the present invention.

[0058] In the pre-training stage of the second training, the intermediate dynamic semantic segmentation model obtained in the previous training is trained with the updated first-category training data to obtain the intermediate dynamic semantic segmentation model again. In the labeling stage of the second training, the second-category training data is labeled with the intermediate dynamic semantic segmentation model obtained in the pre-training stage of this training, and the second-category training data with valid semantic segmentation labels are screened out. The first-category training data is updated based on the second-category training data with valid semantic segmentation labels, thereby increasing the number of training feature data in the first-category training data.

[0059] In the pre-training phase of the third training, the intermediate dynamic semantic segmentation model obtained in the previous training is trained with the updated first-category training data to obtain the intermediate dynamic semantic segmentation model again. In the labeling phase of the second training, the second-category training data is labeled with the intermediate dynamic semantic segmentation model obtained in the pre-training phase of this training, and the second-category training data with valid semantic segmentation labels are screened out. The first-category training data is updated based on the second-category training data with valid semantic segmentation labels, thereby increasing the number of training feature data in the first-category training data.

[0060] Similarly, when the training end condition is reached, the latest intermediate dynamic semantic segmentation model is used as the dynamic semantic segmentation model. The training end condition is set according to actual needs and is not limited in the embodiment of the present invention. For example, the number of training times reaches a preset number of times, and the recognition accuracy of the latest intermediate dynamic semantic segmentation model is greater than a preset value.

[0061] The training method of the dynamic semantic segmentation model provided in the embodiment of the present invention pre-processes the historical video data of the fully-mechanized mining working face of a coal mine to obtain the video sequence of the fully-mechanized mining working face of the coal mine; performs optical flow calculation processing on the video sequence of the fully-mechanized mining working face of the coal mine to obtain the training feature data of the fully-mechanized mining working face of the coal mine; the training feature data includes a first type of training data and a second type of training data, the first type of training data has a corresponding semantic segmentation label, and the second type of training data lacks a corresponding semantic segmentation label; wherein the semantic segmentation label is set based on the fully-mechanized mining working face of the coal mine; according to the first type of training data, the second type of training data and the original model, obtain Dynamic semantic segmentation model; wherein, the dynamic semantic segmentation model is obtained based on the first type of training data, the second type of training data and the original model, including multiple trainings, each training includes a pre-training stage and a labeling stage; the pre-training stage performs model training based on the first type of training data to obtain an intermediate dynamic semantic segmentation model; the labeling stage performs semantic segmentation labeling on the second type of training data based on the intermediate dynamic semantic segmentation model, and screens out the second type of training data with valid semantic segmentation labels to update the first type of training data, which can make full use of video sequences and optical flow information, and improve the accuracy and generalization ability of dynamic semantic segmentation of comprehensive mining working faces. At the same time, since only a small amount of training data needs to be manually labeled to train a dynamic semantic segmentation model, it can reduce the dependence on data labeling and reduce the cost of data labeling, making it easier to train a dynamic semantic segmentation model.

[0062] Figure 2 is a flow chart of a training method for a dynamic semantic segmentation model provided by the second embodiment of the present invention, such as Figure 2 As shown, on the basis of the above embodiments, further, performing optical flow calculation processing on the video sequence of the coal mine fully mechanized mining working face to obtain the training feature data of the coal mine fully mechanized mining working face includes:

[0063] S201, dividing the video sequence into frames to obtain each frame image corresponding to the video sequence;

[0064] Specifically, the server may divide the video sequence into frames to obtain each frame image corresponding to the video sequence. Each frame image corresponding to the video sequence is an image arranged in chronological order.

[0065] S202, for each frame image starting from the second frame image in each frame image corresponding to the video sequence, perform optical flow calculation based on each frame image and a previous frame image corresponding to each frame image to obtain motion information corresponding to each frame image;

[0066] Specifically, the server performs optical flow calculation on each frame image corresponding to the video sequence, that is, starting from the second frame image of each frame image corresponding to the video sequence, optical flow calculation is performed based on each frame image and the previous frame image corresponding to each frame image, and motion information between each frame image and the previous frame image corresponding to each frame image is calculated to obtain motion information corresponding to each frame image. Among them, the optical flow calculation can adopt a traditional optical flow algorithm, such as Lucas-Kanade, etc.; it can also adopt an optical flow algorithm based on deep learning, such as flownet, PWC-Net or RAFT, etc.; it is selected according to actual needs, and the embodiment of the present invention does not limit it.

[0067] S203, obtaining training feature data of the fully mechanized mining face of the coal mine according to the motion information corresponding to each frame image and each frame image.

[0068] Specifically, the server uses the motion information corresponding to each frame of image and each frame of image as a piece of training feature data, and can obtain the training feature data of the coal mine comprehensive mining working face based on the motion information corresponding to each frame of image and each frame of image.

[0069] The present application calculates the motion information of two adjacent frames of images, which can fully capture the dynamic changes in the scene. Using the above motion information as training data is conducive to improving the accuracy of semantic segmentation of the trained dynamic semantic segmentation model.

[0070] Figure 3 is a flow chart of a training method for a dynamic semantic segmentation model provided in the third embodiment of the present invention. Figure 3 As shown, on the basis of the above embodiments, further, the model training is performed based on the first type of training data to obtain the intermediate dynamic semantic segmentation model, including:

[0071] S301, obtaining the latest first-category training data;

[0072] Specifically, during the first training, the first type of training data is the latest, and the first type of training data is directly obtained to train the original model. During the second training and each subsequent training, since the first type of training data is updated, the first type of training data updated in the previous training will be obtained as the latest first type of training data for model training.

[0073] S302: Perform model training according to the latest first-category training data to obtain an intermediate dynamic semantic segmentation model.

[0074] Specifically, during the first training, the original model is trained based on the latest first-category training data to obtain an intermediate dynamic semantic segmentation model. During the second training and each subsequent training, the intermediate dynamic semantic segmentation model obtained from the previous training is retrained based on the latest first-category training data to obtain an intermediate dynamic semantic segmentation model.

[0075] Figure 4 is a flow chart of a training method for a dynamic semantic segmentation model provided by a fourth embodiment of the present invention. Figure 4 As shown, the step of labeling the second type of training data with semantic segmentation labels based on the intermediate dynamic semantic segmentation model, and screening out the second type of training data with valid semantic segmentation labels to update the first type of training data includes:

[0076] S401, according to the second type of training data and the intermediate dynamic semantic segmentation model, obtaining a semantic segmentation prediction and recognition result corresponding to each second training feature data in the second type of training data, wherein the semantic segmentation prediction and recognition result includes a semantic segmentation prediction result;

[0077] Specifically, the server inputs each second training feature data in the second type of training data into the intermediate dynamic semantic segmentation model for semantic segmentation, and can obtain the semantic segmentation prediction and recognition results corresponding to each second training feature data in the second type of training data. The semantic segmentation prediction and recognition results corresponding to the second training feature data may include the semantic segmentation prediction results corresponding to the second training feature data and the area corresponding to the semantic segmentation prediction results corresponding to the second training feature data in the image included in the second training feature data. The semantic segmentation prediction results corresponding to the second training feature data include at least one semantic segmentation prediction result, and the semantic segmentation prediction results may be a support guard plate, a scraper conveyor, a roller, coal, a trough side, a rail or a cable trough, corresponding to a semantic segmentation label. The semantic segmentation prediction results have a corresponding area in the image included in the second training feature data. Each training feature data in the second type of training data includes a frame of image and motion information corresponding to the frame of image.

[0078] S402, determining second training feature data in the second type of training data that has a valid semantic segmentation prediction result according to the semantic segmentation prediction recognition result corresponding to each second training feature data in the second type of training data and a label evaluation rule;

[0079] Specifically, the server evaluates the validity of the semantic segmentation prediction result corresponding to each training feature data in the second category of training data according to the semantic segmentation prediction recognition result corresponding to each training feature data in the second category of training data and the label evaluation rule, and can obtain the training feature data with valid semantic segmentation prediction results in the second category of training data. The label evaluation rule is pre-set.

[0080] S403 , updating the first category of training data according to the second category of training data having valid semantic segmentation prediction results, to obtain updated first category of training data.

[0081] Specifically, after obtaining the training feature data with valid semantic segmentation prediction results in the second type of training data, the server adds the training feature data with valid semantic segmentation prediction results to the original first type of training data to obtain updated first type of training data. The original first type of training data is the first type of training data used in the first training.

[0082] On the basis of the above embodiments, further, the label evaluation rule includes:

[0083] Obtaining a region sequence score corresponding to the second training feature data based on a region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data;

[0084] Based on the region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, obtaining an average region area and a suitable region proportion corresponding to the second training feature data;

[0085] If the region sequence score, average region area and suitable region proportion corresponding to the second training feature data are all greater than their respective thresholds, the semantic segmentation prediction result corresponding to the second training feature data is valid; otherwise, the semantic segmentation prediction result corresponding to the second training feature data is invalid.

[0086] Specifically, in the fully mechanized working face of a coal mine, the installation of objects follows a certain rule, for example, the support guard plate is on the top coal wall, and the scraper conveyor, track, trough side, and cable trough are on the ground in sequence. This order is fixed, and based on this rule, the effectiveness of the semantic segmentation prediction results can be evaluated. For each second training feature data in the second type of training data, the server can evaluate the arrangement order of the area corresponding to the semantic segmentation prediction result corresponding to the second training feature data according to the area corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, and obtain the area order score corresponding to the second training feature data.

[0087] For the region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, the effectiveness can also be evaluated based on the fragmentation degree of each identified region. The server can obtain the average region area and the proportion of suitable regions corresponding to the second training feature data based on the region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, and evaluate the effectiveness of the semantic segmentation prediction result by the above average region area and proportion of suitable regions.

[0088] The server compares the regional order score corresponding to the second training feature data with the corresponding threshold, compares the average regional area corresponding to the second training feature data with the corresponding threshold, and compares the appropriate regional proportion corresponding to the second training feature data with the corresponding threshold. If the regional order score corresponding to the second training feature data is greater than the corresponding threshold, the average regional area corresponding to the second training feature data is greater than the corresponding threshold, and the appropriate regional proportion corresponding to the second training feature data is greater than the corresponding threshold, then the semantic segmentation prediction result corresponding to the second training feature data is valid. Among them, the threshold corresponding to the average regional area, the threshold corresponding to the regional order score, and the threshold corresponding to the appropriate regional proportion are set according to actual experience and are not limited in the embodiments of the present invention.

[0089] If any of the region sequence score corresponding to the second training feature data, the average region area corresponding to the second training feature data, and the suitable region proportion corresponding to the second training feature data is less than the corresponding threshold, then the semantic segmentation prediction result corresponding to the second training feature data is invalid.

[0090] Figure 5 is a flow chart of a training method for a dynamic semantic segmentation model provided in a fifth embodiment of the present invention. Figure 5 As shown, on the basis of the above embodiments, further, the region corresponding to the image included in the second training feature data based on the semantic segmentation prediction result corresponding to the second training feature data, obtaining the region sequence score corresponding to the second training feature data includes:

[0091] S501, based on the semantic segmentation prediction result corresponding to the second training feature data, the area corresponding to the image included in the second training feature data is obtained, the minimum bounding rectangle of the area corresponding to each set semantic segmentation label in the image;

[0092] Specifically, there may be one or more semantic segmentation prediction results corresponding to the second training feature data, and the semantic segmentation prediction results corresponding to the second training feature data have corresponding areas in the image included in the second training feature data. The server can obtain the minimum circumscribed rectangle of the area in the image corresponding to each set semantic segmentation label corresponding to the second training feature data based on the area in the image included in the second training feature data corresponding to each set semantic segmentation label corresponding to the second training feature data. Among them, each set semantic segmentation label is selected from the semantic segmentation prediction results, and each set semantic segmentation label may include a bracket guard plate, a cable groove, and a groove side. The area corresponding to each set semantic segmentation label in the image should be a long strip.

[0093] S502, obtaining each reserved semantic segmentation label based on the size of the minimum bounding rectangle of the area corresponding to each set semantic segmentation label in the image;

[0094] Specifically, the server screens the areas corresponding to each set semantic segmentation label in the image according to the size of the minimum circumscribed rectangle of the area corresponding to each set semantic segmentation label in the image, obtains outlier samples from the areas corresponding to each set semantic segmentation label in the image, and removes the outlier semantic segmentation labels from the semantic segmentation prediction results corresponding to the second training feature data as outlier semantic segmentation labels to obtain each retained semantic segmentation label.

[0095] For example, the length and width of the minimum bounding rectangle of the area corresponding to the set semantic segmentation label in the image of the semantic segmentation model can be used to calculate the ratio of length to width. If the ratio of the length to width is less than 2:1, the area corresponding to the set semantic segmentation label in the image is regarded as an outlier sample. If the ratio of the length to width is greater than or equal to 2:1, the set semantic segmentation label is regarded as a reserved semantic segmentation label.

[0096] S503, based on the size of the minimum bounding rectangle of the area corresponding to each retained semantic segmentation label in the image and the area weight calculation formula, calculate and obtain the area weight corresponding to each retained semantic segmentation label; wherein the area weight calculation formula is preset;

[0097] Specifically, the size of the minimum bounding rectangle of the area corresponding to each retained semantic segmentation label in the image is brought into the area weight calculation formula, and the area weight corresponding to each retained semantic segmentation label can be calculated.

[0098] For example, the regional weight calculation formula is: Power_area i Area represents the area weight corresponding to the i-th retained semantic segmentation label.i represents the area of ​​the region corresponding to the i-th retained semantic segmentation label in the image, w represents the width of the image corresponding to the second training feature data, h represents the length of the image corresponding to the second training feature data, and w i represents the width of the minimum bounding rectangle of the region corresponding to the i-th preserved semantic segmentation label in the image, h i represents the length of the minimum bounding rectangle of the area corresponding to the i-th preserved semantic segmentation label in the image.

[0099] S504, obtaining a comprehensive weight corresponding to each retained semantic segmentation label according to the region weight and position weight corresponding to each retained semantic segmentation label; wherein the position weight corresponding to each retained semantic segmentation label is preset;

[0100] Specifically, the server may obtain a comprehensive weight corresponding to each retained semantic segmentation label according to the region weight corresponding to each retained semantic segmentation label and the position weight corresponding to each retained semantic segmentation label. The position weight corresponding to each retained semantic segmentation label is preset and is set according to actual needs, and is not limited in the embodiment of the present invention.

[0101] For example, the sum of the region weight corresponding to the retained semantic segmentation label and the position weight corresponding to the retained semantic segmentation label is calculated to obtain the comprehensive weight corresponding to the retained semantic segmentation label.

[0102] For example, the position weights corresponding to the semantic segmentation labels bracket guard plate, scraper conveyor, roller, coal, trough side, track, and cable trough are retained as 1, 2, 3, 4, 5, 6, and 7 respectively.

[0103] S505, based on the angle between the long side of the minimum circumscribed rectangle of the area corresponding to each retained semantic segmentation label in the image and the horizontal direction of the image and the center of the image corresponding to the second training feature data, obtain the image direction corresponding to the second training feature data and obtain the vertical line of the image direction corresponding to the training feature data;

[0104] Specifically, the server can obtain the angle between the long side of the minimum bounding rectangle of the area corresponding to each retained semantic segmentation label in the image and the horizontal direction of the image included in the second training feature data, average the angles between the long side of the minimum bounding rectangle of the area corresponding to each retained semantic segmentation label in the image and the horizontal direction of the image included in the second training feature data, and use the obtained average angle as the image direction corresponding to the second training feature data. Based on the image direction corresponding to the second training feature data and the center of the image corresponding to the second training feature data, a vertical line of the image direction corresponding to the second training feature data can be obtained.

[0105] The image in which the semantic segmentation label is retained in the region corresponding to the image refers to the image including the training feature data corresponding to the semantic segmentation label. The long side of the minimum bounding rectangle refers to the side where the length of the minimum bounding rectangle is located.

[0106] S506, projecting the center of the region corresponding to each retained semantic segmentation label in the image onto a vertical line in the image direction corresponding to the second training feature data, to obtain the coordinates of the projection point corresponding to each retained semantic segmentation label;

[0107] Specifically, the region corresponding to each retained semantic segmentation label in the image has a region center, that is, the region center corresponding to each retained semantic segmentation label in the image. The region center corresponding to each retained semantic segmentation label in the image is projected onto a vertical line in the image direction corresponding to the training feature data, and the coordinates of the projection point corresponding to each retained semantic segmentation label can be obtained.

[0108] S507, obtaining a deduction score corresponding to the second training feature data based on the projection point coordinates, comprehensive weights, and regional weights corresponding to each retained semantic segmentation label;

[0109] Specifically, the server can obtain the deduction points corresponding to the second training feature data according to the projection point coordinates, comprehensive weights and regional weights corresponding to each retained semantic segmentation label. The specific implementation process is described below and will not be repeated here.

[0110] S508: Calculate the difference between the preset full score and the deduction score corresponding to the second training feature data to obtain the regional sequence score corresponding to the second training feature data.

[0111] Specifically, the server calculates a difference between a preset full score and a deduction score corresponding to the second training feature data, and uses the difference as a regional sequence score corresponding to the second training feature data.

[0112] Figure 6 is a flow chart of a training method for a dynamic semantic segmentation model provided by a sixth embodiment of the present invention, such as Figure 6 As shown, on the basis of the above embodiments, further, the deduction points corresponding to the second training feature data are obtained based on the projection point coordinates, comprehensive weights and regional weights corresponding to each retained semantic segmentation label, including:

[0113] S601, sorting the projection point coordinates corresponding to each retained semantic segmentation label in order from small to large, to obtain a first arrangement order of each retained semantic segmentation label; and sorting the projection point coordinates corresponding to each retained semantic segmentation label in order from large to small, to obtain a second arrangement order of each retained semantic segmentation label;

[0114] Specifically, the server arranges the projected point coordinates corresponding to the retained semantic segmentation labels in order from small to large, and obtains a first arrangement order of the retained semantic segmentation labels. The server arranges the projected point coordinates corresponding to the retained semantic segmentation labels in order from large to small, and obtains a second arrangement order of the retained semantic segmentation labels.

[0115] S602, obtaining a first order deduction score corresponding to the second training feature data according to the first arrangement order and the comprehensive weight and regional weight of the retained semantic segmentation labels that are adjacent to the order; and obtaining a second order deduction score corresponding to the second training feature data according to the second arrangement order and the comprehensive weight and regional weight of the retained semantic segmentation labels that are adjacent to the order;

[0116] Specifically, the server obtains the first retained semantic segmentation label as the current retained semantic segmentation label in the first arrangement order, compares the comprehensive weight corresponding to the current retained semantic segmentation label with the comprehensive weight corresponding to the next retained semantic segmentation label of the current retained semantic segmentation label, and if the comprehensive weight corresponding to the current retained semantic segmentation label is greater than the comprehensive weight corresponding to the next retained semantic segmentation label of the current retained semantic segmentation label, then the regional weight corresponding to the current retained semantic segmentation label is continued to be compared with the regional weight corresponding to the next retained semantic segmentation label of the current retained semantic segmentation label. If the comprehensive weight corresponding to the current retained semantic segmentation label is not less than the comprehensive weight corresponding to the next retained semantic segmentation label of the current retained semantic segmentation label, then the next retained semantic segmentation label of the current retained semantic segmentation label is continued to be obtained as the current retained semantic segmentation label, and the above-mentioned comprehensive weight comparison is performed again.

[0117] If the regional weight corresponding to the current retained semantic segmentation label is less than the regional weight corresponding to the next retained semantic segmentation label of the current retained semantic segmentation label, then the first order deduction score corresponding to the second training feature data is updated to the regional weight corresponding to the current retained semantic segmentation label, and the next retained semantic segmentation label of the current retained semantic segmentation label is obtained as the current retained semantic segmentation label, and the above comprehensive weight comparison is performed again. If the regional weight corresponding to the current retained semantic segmentation label is not less than the regional weight corresponding to the next retained semantic segmentation label of the current retained semantic segmentation label, then the first order deduction score corresponding to the second training feature data is updated to the regional weight corresponding to the next retained semantic segmentation label of the current retained semantic segmentation label, and the next retained semantic segmentation label of the current retained semantic segmentation label is skipped, and the next retained semantic segmentation label of the current retained semantic segmentation label is obtained as the current retained semantic segmentation label, and the above comprehensive weight comparison is performed again.

[0118] Repeat the above processing of the currently reserved semantic segmentation label until no reserved semantic segmentation label can be used as the currently reserved semantic segmentation label. The initial value of the first order deduction score corresponding to the second training feature data is equal to 0.

[0119] The specific implementation process of obtaining the second order deduction points corresponding to the second training feature data according to the second arrangement order and the comprehensive weights and regional weights of the retained semantic segmentation labels that are adjacent to the order is similar to the specific implementation process of obtaining the first order deduction points corresponding to the second training feature data according to the first arrangement order and the comprehensive weights and regional weights of the retained semantic segmentation labels that are adjacent to the order, and will not be repeated here. The initial value of the second order deduction points corresponding to the second training feature data is equal to 0.

[0120] The following is a specific example to illustrate the specific process of obtaining the first order deduction points corresponding to the second training feature data. Figure 7 shown.

[0121] The first step is to sort the projection point coordinates. The server arranges the projection point coordinates corresponding to each retained semantic segmentation label in ascending order according to the size of the projection point coordinates, and obtains the first arrangement order of each retained semantic segmentation label, and obtains the number Q of retained semantic segmentation labels in the first arrangement order.

[0122] Step 2: Initialize relevant data: Set i=1, and the first order deduction score P corresponding to the second training feature data=0.

[0123] Step 3: Determine whether i is less than Q. If i is less than Q, go to step 4; if i is greater than or equal to Q, go to step 9.

[0124] Step 4: Determine a i Is it greater than a i+1 If the comprehensive weight a corresponding to the i-th retained semantic segmentation label in the first arrangement order i Greater than the comprehensive weight a corresponding to the i-th retained semantic segmentation label in the first arrangement order i+1 , then go to step 5; if the comprehensive weight a corresponding to the i-th retained semantic segmentation label in the first arrangement order i Less than or equal to the comprehensive weight a corresponding to the i-th retained semantic segmentation label in the first arrangement order i+1 , then proceed to step 8.

[0125] Step 5: Determine b i Is it less than b? i+1 If the comprehensive weight b corresponding to the i-th retained semantic segmentation label in the first arrangement order i Less than the comprehensive weight b corresponding to the i-th retained semantic segmentation label in the first sorting orderi+1 , then go to step 6; if the comprehensive weight b corresponding to the i-th retained semantic segmentation label in the first arrangement order i Greater than or equal to the comprehensive weight b corresponding to the i-th retained semantic segmentation label in the first arrangement order i+1 , then proceed to step 7.

[0126] Step 6: Update the P value. Calculate P = P + b i .

[0127] Step 7: Update the P value. Calculate P = P + b i+1 , and remove the i+1th retained semantic segmentation label from the first permutation order.

[0128] Step 8. Add 1 to the value of i. Calculate i=i+1.

[0129] Step 9. Output the P value.

[0130] S603: Obtain a smaller value between the first order deduction points corresponding to the second training feature data and the second order deduction points corresponding to the second training feature data as the deduction points corresponding to the second training feature data.

[0131] Specifically, the server compares the first order deduction points corresponding to the second training feature data and the second order deduction points corresponding to the second training feature data, and uses the smaller value of the two as the deduction points corresponding to the second training feature data.

[0132] Figure 8 is a flow chart of a training method for a dynamic semantic segmentation model provided in the eighth embodiment of the present invention, such as Figure 8 As shown, on the basis of the above embodiments, further, the region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, obtaining the average region area and the appropriate region proportion corresponding to the second training feature data includes:

[0133] S801, according to the area corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, obtaining the area of ​​the area corresponding to each semantic segmentation prediction result in the image; and counting the number of semantic segmentation prediction results corresponding to the second training feature data to obtain the total number of areas corresponding to the second training feature data;

[0134] Specifically, there may be one or more semantic segmentation prediction results corresponding to the second training feature data, and the semantic segmentation prediction results corresponding to the second training feature data have corresponding areas in the image included in the second training feature data. The server may obtain the area of ​​the region corresponding to each semantic segmentation prediction result in the image based on the area corresponding to the semantic segmentation prediction results corresponding to the second training feature data in the image included in the second training feature data. The server may count the number of semantic segmentation prediction results corresponding to the second training feature data as the total number of areas corresponding to the second training feature data.

[0135] S802, obtaining the total area of ​​the region corresponding to the second training feature data according to the area of ​​the region corresponding to each semantic segmentation prediction result in the image;

[0136] Specifically, the server calculates the sum of the area of ​​the region corresponding to each semantic segmentation prediction result in the image in the semantic segmentation prediction result corresponding to the second training feature data, and obtains the total area of ​​the region corresponding to the second training feature data.

[0137] S803, obtaining an average area of ​​regions corresponding to the second training feature data according to the total number of regions and the total area of ​​regions corresponding to the second training feature data;

[0138] Specifically, the server calculates a quotient of a total area of ​​regions corresponding to the second training feature data divided by a total number of regions corresponding to the second training feature data to obtain an average area of ​​regions corresponding to the second training feature data.

[0139] S804, obtaining the number of suitable regions corresponding to the second training feature data according to the area of ​​the region corresponding to each semantic segmentation prediction result in the image and the area threshold;

[0140] Specifically, the server compares the area of ​​the region corresponding to each semantic segmentation prediction result in the image with the area threshold, counts the number of regions whose area is greater than the area threshold, and obtains the number of suitable regions corresponding to the second training feature data. The area threshold is set according to actual needs and is not limited in the embodiment of the present invention.

[0141] S805 . Obtain a proportion of suitable regions corresponding to the second training feature data according to the number of suitable regions corresponding to the second training feature data and the total number of regions corresponding to the second training feature data.

[0142] Specifically, the server calculates the ratio of the number of suitable regions corresponding to the second training feature data to the total number of regions corresponding to the second training feature data to obtain the proportion of suitable regions corresponding to the second training feature data.

[0143] Fig. 9is a flow chart of a training method for a dynamic semantic segmentation model provided by the ninth embodiment of the present invention, such as Fig. 9 As shown, on the basis of the above embodiments, further, the model training is performed based on the first type of training data to obtain the intermediate dynamic semantic segmentation model, including:

[0144] S901, calculating a first loss value corresponding to first training feature data in the first type of training data based on a first loss function;

[0145] Specifically, during the training process of the intermediate dynamic semantic segmentation model, for each first training feature data in the first category of training data, the server may calculate a first loss value corresponding to each first training feature data according to a first loss function, wherein the first loss function is preset.

[0146] During the training process of the intermediate dynamic semantic segmentation model, the first training feature data is input into the training model, and each semantic segmentation prediction result corresponding to the first training feature data can be output, and each semantic segmentation label in the semantic segmentation label corresponding to the first training feature data is used as a true semantic segmentation label, and the error between the semantic segmentation prediction result and the corresponding true semantic segmentation label can be calculated through the first loss function. The first loss function can adopt Dice loss function, cross entropy loss function, etc., and is selected according to actual needs, and the embodiment of the present invention does not limit it.

[0147] S902, calculating a second loss value corresponding to the first training feature data in the first type of training data based on a second loss function;

[0148] Specifically, the second loss function is designed based on the application scenario of the present application, and can be obtained by calculating the ratio of the number of pixels of the areas corresponding to other semantic segmentation prediction results included in the areas corresponding to the semantic segmentation prediction results corresponding to the first training feature data to the total number of pixels of the image corresponding to the first training feature data. The second loss function can reduce the phenomenon of identifying that the areas corresponding to the semantic segmentation prediction results corresponding to the first training feature data include other areas. Wherein, the second loss function is pre-set.

[0149] S903, calculating a third loss value corresponding to the first training feature data in the first type of training data based on a third loss function;

[0150] Specifically, during the training process of the intermediate dynamic semantic segmentation model, for each first training feature data in the first category of training data, the server can calculate the loss value corresponding to each semantic segmentation prediction result corresponding to each first training feature data according to the first loss function, and then normalize the loss value corresponding to each semantic segmentation prediction result to obtain the normalized loss value corresponding to each semantic segmentation prediction result. Then calculate the sum of the normalized loss value corresponding to each semantic segmentation prediction result and the product of their corresponding weights as the third loss value corresponding to the first training feature data. Wherein, the third loss function is pre-set.

[0151] Since there are different types of semantic segmentation labels such as support guard plates, scraper conveyors, rollers, coal, trough sides, rails, cable troughs, etc. in the comprehensive mining working face of a coal mine, the numerical range of the loss values ​​corresponding to different semantic segmentation labels is different. In order to eliminate the dimensional difference and make the loss values ​​corresponding to different semantic segmentation labels have equal importance and influence in model training, the server normalizes the loss value corresponding to each semantic segmentation prediction result corresponding to each first training feature data, and the normalized loss value of each semantic segmentation prediction result of each first training data can be obtained. Since the number of samples corresponding to each semantic segmentation label in the training sample is different, there is a low recognition accuracy due to the small number of samples and unclear features. By assigning different weights to different semantic segmentation labels, semantic segmentation labels with a small number of samples and difficult to identify are assigned larger weights, which is conducive to improving the recognition accuracy of semantic segmentation labels.

[0152] For example, for the loss values ​​corresponding to the various semantic segmentation prediction results, there may be a situation where the loss value corresponding to a certain semantic segmentation prediction result is particularly large, resulting in large fluctuations in the model loss, and also causing the loss values ​​corresponding to other semantic segmentation prediction results to appear relatively small, which is not conducive to model learning. In the process of calculating the loss value corresponding to the first training feature data in the pre-training stage, the loss value of each semantic segmentation prediction result can be normalized by the sigmoid function. The loss value of each semantic segmentation prediction result of each first training feature data is normalized to obtain the normalized loss value of each semantic segmentation prediction result of each first training feature data, including:

[0153] By formula Calculate the normalized loss value corresponding to the semantic segmentation prediction result. X′ represents the normalized loss value corresponding to the semantic segmentation prediction result. X represents the loss value corresponding to the semantic segmentation prediction result. max is a constant, a is a constant, and b is a constant. For example, max is 10, a is 0.5, and b is 2.

[0154] S904. Sum the first loss value, the second loss value, and the third loss value to obtain a total loss value corresponding to the first training feature data.

[0155] Specifically, after calculating and obtaining the first loss value, the second loss value, and the third loss value, the sum of the first loss value, the second loss value, and the third loss value is calculated, and the sum is used as the total loss value corresponding to the first training feature data. After obtaining the total loss value corresponding to the first training feature data, the parameters of the trained intermediate dynamic semantic segmentation model can be adjusted with reference to the methods of the prior art, for example, the weights of the intermediate dynamic semantic segmentation model can be adjusted by back propagation, gradient descent, and other methods.

[0156] On the basis of the above embodiments, further, the second loss function is:

[0157]

[0158] Among them, loss c Indicates the first loss value corresponding to the first training feature data, num c It represents the number of pixels of the area corresponding to other semantic segmentation prediction results included in the area corresponding to the semantic segmentation prediction result corresponding to the first training feature data, W represents the first number of pixels corresponding to the first training feature data, and H represents the second number of pixels corresponding to the first training feature data.

[0159] Specifically, the semantic segmentation prediction result corresponding to the first training feature data has a corresponding area in the image included in the first training feature data, that is, the area corresponding to the semantic segmentation prediction result. The minimum bounding rectangle of the image corresponding to the first training feature data is obtained, the number of pixels of the minimum bounding rectangle in the length direction is obtained as the first number of pixels corresponding to the first training feature data, and the number of pixels of the minimum bounding rectangle in the width direction is obtained as the second number of pixels corresponding to the first training feature data.

[0160] The number of pixels num in the area corresponding to other semantic segmentation prediction results included in the area corresponding to the semantic segmentation prediction result corresponding to the first training feature data c The specific calculation process is as follows:

[0161] (1) traversing all regions corresponding to the semantic segmentation prediction results in the image corresponding to the first training feature data;

[0162] (2) For each predicted area, calculate its outer contour coordinates;

[0163] (3) For each prediction region: create a new array of all zeros, denoted as array new, whose size is (W, H), which is the same as the minimum bounding rectangle size of the image corresponding to the first training feature data. Then, according to the outer contour coordinates of the prediction area, in the newly created array array new Fill it up and mark the inner area of ​​the contour as 1;

[0164] (4) After the traversal is completed, all arrays generated by (3) new Add together to get a cumulative array array coincide ;

[0165] (5) In the cumulative array array coincide , the number of elements whose statistical value is greater than 1. This value is the total number of pixels with overlapping regions in all semantic segmentation predictions, that is, the number of pixels with nested labels in the semantic segmentation prediction results to be solved, recorded as num c .

[0166] By calculating num c , we can quantify the degree of nested labels in the prediction results and provide a basis for subsequent optimization.

[0167] On the basis of the above embodiments, further, the third loss function is:

[0168]

[0169] Among them, loss represents the third loss value corresponding to the first training feature data, loss i represents the normalized loss value corresponding to the i-th semantic segmentation prediction result corresponding to the first training feature data, I i Represents the intersection-over-union ratio corresponding to the i-th semantic segmentation prediction result.

[0170] Specifically, the area of ​​the i-th semantic segmentation prediction result in the picture included in the first training feature data is the prediction area corresponding to the i-th semantic segmentation prediction result, and the area of ​​the real semantic segmentation label corresponding to the i-th semantic segmentation prediction result in the picture included in the first training feature data is the real area corresponding to the i-th semantic segmentation prediction result. Calculate the ratio of the intersection and union of the prediction area corresponding to the i-th semantic segmentation prediction result and the real area corresponding to the i-th semantic segmentation prediction result to obtain the intersection-union ratio corresponding to the i-th semantic segmentation prediction result. Calculate the sum of the products of the normalized loss value of each semantic segmentation prediction result and the intersection-union ratio corresponding to each corresponding semantic segmentation prediction result to obtain the third loss value corresponding to the first training feature data.

[0171] The loss value corresponding to the i-th semantic segmentation prediction result corresponding to the first training feature data can be calculated by establishing a formula by comprehensively considering the IOU, area relationship, perimeter relationship, center point distance, area number difference, and number of areas with too small areas in the prediction result between the predicted area corresponding to the i-th semantic segmentation prediction result and the corresponding real area. The loss value corresponding to the i-th semantic segmentation prediction result corresponding to the first training feature data can be calculated by the following formula.

[0172] loss i =L1+L2+L3+L4+L5+L6

[0173] Among them, L1 is equal to the intersection-over-union ratio of the predicted area corresponding to the i-th semantic segmentation prediction result and the corresponding true area. A1 is the area of ​​the predicted region corresponding to the i-th semantic segmentation prediction result, and A2 is the area of ​​the true region corresponding to the i-th semantic segmentation prediction result; C1 represents the perimeter of the predicted area corresponding to the i-th semantic segmentation prediction result, and C2 represents the perimeter of the real area corresponding to the i-th semantic segmentation prediction result. (x2, y2) represents the center point coordinates of the true area corresponding to the i-th semantic segmentation prediction result, and (x1, y1) represents the center point coordinates of the predicted area corresponding to the i-th semantic segmentation prediction result. a2 represents the number of real regions corresponding to the i-th semantic segmentation prediction result, and a1 represents the number of pre-regions corresponding to the i-th semantic segmentation prediction result. e is a constant, Q is a constant, and they are selected based on practical experience, for example, e is 10 and Q is 100.

[0174] The semantic segmentation of the coal mine fully mechanized working face of the present application is different from the focus of semantic segmentation in the prior art. The semantic segmentation in the prior art, that is, classifying each pixel, only pays attention to the classification results of the region from the pixel level. However, in the present application, more attention is paid to the recognition of the region. If only the accuracy at the pixel level is paid attention to in the prior art, some recognition errors will be caused. For example: if the background of 10 pixels is mistakenly identified as the coal machine drum in a scene without a coal machine drum, if only the pixels are paid attention to, the error seems insignificant, but in the present application, the coal machine drum will be mistakenly identified, which will have a negative impact on subsequent work. Similarly, if the coal machine drum is mistakenly identified as 10 pixels more in a scene with a coal machine drum, if only the accuracy at the pixel level is paid attention to, the loss value is the same as the above scene, but for the present application, the impact of such errors is negligible. In addition, the present application gives priority to the misidentification and missed recognition of the region, and secondly pays attention to the position offset between the recognition area and the actual area, the inconsistency between the size of the recognition area and the actual area, etc. Therefore, the loss function is redesigned based on the application scenario of the present application, and good results are achieved.

[0175] The following describes the specific implementation process of the semantic segmentation method for coal mine fully mechanized working face provided by the embodiment of the present invention by taking the server as the execution subject. It is understandable that the execution subject of the semantic segmentation method for coal mine fully mechanized working face provided by the embodiment of the present invention is not limited to the server.

[0176] Fig.10 is a flow chart of a semantic segmentation method for a fully mechanized coal mining face provided by the tenth embodiment of the present invention, such as Fig.10 As shown, an embodiment of the present invention provides a semantic segmentation method for a fully mechanized coal mining face, which adopts the dynamic semantic segmentation model described in any of the above embodiments, including:

[0177] S1001, obtaining current frame video data based on video data of a fully mechanized mining face in a coal mine;

[0178] Specifically, when coal is mined underground using coal mining equipment, a camera will be installed to collect video data of the coal mine's comprehensive mining working face. The camera can send the video data of the coal mine's comprehensive mining working face to the server. The server will receive the video data of the coal mine's comprehensive mining working face and obtain the current frame video data from it. The current frame video data is a frame of video data currently being processed by the server.

[0179] S1002, preprocessing the current frame video data to obtain a current frame image corresponding to the current frame video data;

[0180] Specifically, the server preprocesses the current frame video data to obtain the current frame image corresponding to the current frame video data. The preprocessing includes but is not limited to format conversion, normalization processing, etc., which are set according to actual needs and are not limited in the embodiment of the present invention.

[0181] S1003, performing optical flow calculation processing on a current frame image corresponding to the current frame video data and a previous frame image of the current frame image corresponding to the current frame video data to obtain predicted feature data corresponding to the current frame video data;

[0182] Specifically, the server performs optical flow calculation processing on the current frame image corresponding to the current frame video data and the previous frame image of the current frame image corresponding to the current frame video data, and can obtain the predicted feature data corresponding to the current frame video data. The predicted feature data corresponding to the current frame video data includes the current frame image and the motion information between the current frame image and the previous frame image corresponding to the current frame image.

[0183] S1004: Obtain a semantic segmentation result corresponding to the current frame video data according to the predicted feature data corresponding to the current frame video data and a dynamic semantic segmentation model.

[0184] Specifically, the server inputs the predicted feature data corresponding to the current frame video data into a dynamic semantic segmentation model, performs semantic segmentation on the current frame image, and obtains a semantic segmentation result corresponding to the current frame video data. The semantic segmentation result corresponding to the current frame video data includes a semantic segmentation label corresponding to the current frame image.

[0185] The semantic segmentation method for a coal mine fully mechanized mining working face provided in an embodiment of the present invention obtains current frame video data based on video stream data of the coal mine fully mechanized mining working face; pre-processes the current frame video data to obtain a current frame image corresponding to the current frame video data; performs optical flow calculation processing on the current frame image corresponding to the current frame video data and the previous frame image of the current frame image corresponding to the current frame video data to obtain predicted feature data corresponding to the current frame video data; obtains a semantic segmentation result corresponding to the current frame video data based on the predicted feature data corresponding to the current frame video data and a dynamic semantic segmentation model, thereby improving the accuracy of dynamic semantic segmentation of the fully mechanized mining working face.

[0186] Fig.11 is a schematic diagram of the structure of a training device for a dynamic semantic segmentation model provided by the eleventh embodiment of the present invention. Fig.11 As shown, the training device of the dynamic semantic segmentation model provided by the embodiment of the present invention is applied to a fully mechanized mining face in a coal mine, and includes a preprocessing unit 1101, an optical flow calculation processing unit 1102 and a training unit 1103, wherein:

[0187] The preprocessing unit 1101 is used to preprocess the historical video data of the coal mine fully mechanized mining working face to obtain the video sequence of the coal mine fully mechanized mining working face; the optical flow calculation processing unit 1102 is used to perform optical flow calculation processing on the video sequence of the coal mine fully mechanized mining working face to obtain the training feature data of the coal mine fully mechanized mining working face; the training feature data includes a first type of training data and a second type of training data, the first type of training data has a corresponding semantic segmentation label, and the second type of training data lacks a corresponding semantic segmentation label; wherein the semantic segmentation label is set based on the coal mine fully mechanized mining working face; the training unit 1103 is used to obtain a dynamic semantic segmentation model according to the first type of training data, the second type of training data and the original model; wherein, obtaining the dynamic semantic segmentation model according to the first type of training data, the second type of training data and the original model includes multiple trainings, each training includes a pre-training stage and a labeling stage; the pre-training stage performs model training based on the first type of training data to obtain an intermediate dynamic semantic segmentation model; the labeling stage performs semantic segmentation label annotation on the second type of training data based on the intermediate dynamic semantic segmentation model, and screens out the second type of training data with valid semantic segmentation labels to update the first type of training data.

[0188] Specifically, historical video data of a fully-mechanized coal mining face can be collected, and the preprocessing unit 1101 can preprocess the historical video data of the fully-mechanized coal mining face to obtain a video sequence of the fully-mechanized coal mining face, where the video sequence includes a series of continuous images. The preprocessing includes but is not limited to format conversion, normalization processing, etc., which are set according to actual needs and are not limited in the embodiments of the present invention.

[0189] The optical flow calculation processing unit 1102 performs optical flow calculation on the video sequence of the coal mine fully-mechanized mining working face, obtains the motion information between each frame image and the previous frame image in the video sequence of the coal mine fully-mechanized mining working face, and uses the motion information between each frame image and the previous frame image and each frame image as a training feature data, and the motion information between each frame image and the previous frame image and each frame image constitute the training feature data of the coal mine fully-mechanized mining working face. The training feature data of the coal mine fully-mechanized mining working face includes a first type of training data and a second type of training data. The first type of training data is manually annotated and has corresponding semantic segmentation labels, and the second type of training data lacks corresponding semantic segmentation labels. The semantic segmentation labels are set based on the coal mine fully-mechanized mining working face, and may include categories such as support guard plates, scraper conveyors, rollers, coal, trough sides, rails, cable troughs, etc., and are set according to actual needs, and the embodiments of the present invention do not limit them.

[0190] Since the second category of training data does not have corresponding semantic segmentation labels and cannot be directly used for model training, in the pre-training stage of the first training, the original model is first trained with the first category of training data with semantic segmentation labels to obtain an intermediate dynamic semantic segmentation model. In the labeling stage of the first training, the training unit 1103 labels the second category of training data with the intermediate dynamic semantic segmentation model obtained in the pre-training stage, and screens out the second category of training data with valid semantic segmentation labels. The first category of training data is updated based on the second category of training data with valid semantic segmentation labels, thereby increasing the number of training feature data included in the first category of training data. Among them, the original model can adopt the DeeplabV3+ model, which is selected according to actual needs and is not limited in the embodiments of the present invention.

[0191] In the pre-training stage of the second training, the intermediate dynamic semantic segmentation model obtained in the previous training is trained with the updated first-category training data to obtain the intermediate dynamic semantic segmentation model again. In the labeling stage of the second training, the second-category training data is labeled with the intermediate dynamic semantic segmentation model obtained in the pre-training stage of this training, and the second-category training data with valid semantic segmentation labels are screened out. The first-category training data is updated based on the second-category training data with valid semantic segmentation labels, thereby increasing the number of training feature data in the first-category training data.

[0192] In the pre-training phase of the third training, the intermediate dynamic semantic segmentation model obtained in the previous training is trained with the updated first-category training data to obtain the intermediate dynamic semantic segmentation model again. In the labeling phase of the second training, the second-category training data is labeled with the intermediate dynamic semantic segmentation model obtained in the pre-training phase of this training, and the second-category training data with valid semantic segmentation labels are screened out. The first-category training data is updated based on the second-category training data with valid semantic segmentation labels, thereby increasing the number of training feature data in the first-category training data.

[0193] Similarly, when the training end condition is reached, the latest intermediate dynamic semantic segmentation model is used as the dynamic semantic segmentation model. The training end condition is set according to actual needs and is not limited in the embodiment of the present invention. For example, the number of training times reaches a preset number of times, and the recognition accuracy of the latest intermediate dynamic semantic segmentation model is greater than a preset value.

[0194] The training device of the dynamic semantic segmentation model provided in the embodiment of the present invention pre-processes the historical video data of the fully-mechanized mining working face of a coal mine to obtain the video sequence of the fully-mechanized mining working face of the coal mine; performs optical flow calculation processing on the video sequence of the fully-mechanized mining working face of the coal mine to obtain the training feature data of the fully-mechanized mining working face of the coal mine; the training feature data includes a first type of training data and a second type of training data, the first type of training data has a corresponding semantic segmentation label, and the second type of training data lacks a corresponding semantic segmentation label; wherein the semantic segmentation label is set based on the fully-mechanized mining working face of the coal mine; according to the first type of training data, the second type of training data and the original model, obtain Dynamic semantic segmentation model; wherein, the dynamic semantic segmentation model is obtained based on the first type of training data, the second type of training data and the original model, including multiple trainings, each training includes a pre-training stage and a labeling stage; the pre-training stage performs model training based on the first type of training data to obtain an intermediate dynamic semantic segmentation model; the labeling stage performs semantic segmentation labeling on the second type of training data based on the intermediate dynamic semantic segmentation model, and screens out the second type of training data with valid semantic segmentation labels to update the first type of training data, which can make full use of video sequences and optical flow information, and improve the accuracy and generalization ability of dynamic semantic segmentation of comprehensive mining working faces. At the same time, since only a small amount of training data needs to be manually labeled to train a dynamic semantic segmentation model, it can reduce the dependence on data labeling and reduce the cost of data labeling, making it easier to train a dynamic semantic segmentation model.

[0195] Fig.12 is a schematic diagram of the structure of a training device for a dynamic semantic segmentation model provided by the twelfth embodiment of the present invention. Fig.12 As shown, the training unit 1103 includes a frame division subunit 110301, an optical flow calculation subunit 110302 and an acquisition subunit 110303, wherein:

[0196] The frame division subunit 110301 is used to divide the video sequence into frames to obtain each frame image corresponding to the video sequence; the optical flow calculation subunit 110302 is used to perform optical flow calculation on each frame image starting from the second frame image in each frame image corresponding to the video sequence, based on each frame image and the previous frame image corresponding to each frame image, to obtain the motion information corresponding to each frame image; the acquisition subunit 110303 is used to obtain the training feature data of the coal mine comprehensive mining working face according to the motion information corresponding to each frame image and each frame image.

[0197] Fig.13 is a schematic diagram of the structure of a training device for a dynamic semantic segmentation model provided by the thirteenth embodiment of the present invention. Fig.13As shown, based on the above embodiments, the training unit 1103 further includes an acquisition subunit 110304 and a training subunit 110305, wherein:

[0198] The acquisition subunit 110304 is used to obtain the latest first-category training data; the training subunit 110305 is used to perform model training based on the latest first-category training data to obtain an intermediate dynamic semantic segmentation model.

[0199] Fig.14 is a schematic diagram of the structure of a training device for a dynamic semantic segmentation model provided in the fourteenth embodiment of the present invention. Fig.14 As shown, based on the above embodiments, further, the training unit 1103 includes a prediction subunit 110306, a determination subunit 110307 and an update subunit 110308, wherein:

[0200] The prediction subunit 110306 is used to obtain the semantic segmentation prediction and recognition results corresponding to each second training feature data in the second category of training data according to the second category of training data and the intermediate dynamic semantic segmentation model, and the semantic segmentation prediction and recognition results include semantic segmentation prediction results; the determination subunit 110307 is used to determine the second training feature data with valid semantic segmentation prediction results in the second category of training data according to the semantic segmentation prediction and recognition results corresponding to each second training feature data in the second category of training data and the label evaluation rules; the updating subunit 110308 is used to update the first category of training data according to the second training feature data with valid semantic segmentation prediction results in the second category of training data, so as to obtain updated first category training data.

[0201] On the basis of the above embodiments, further, the label evaluation rule includes:

[0202] Based on the area corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, the area sequence score corresponding to the second training feature data is obtained; based on the area corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, the average area of ​​the region and the proportion of suitable regions corresponding to the second training feature data are obtained; if the area sequence score, the average area of ​​the region and the proportion of suitable regions corresponding to the second training feature data are all greater than their respective corresponding thresholds, the semantic segmentation prediction result corresponding to the second training feature data is valid; otherwise, the semantic segmentation prediction result corresponding to the second training feature data is invalid.

[0203] On the basis of the above embodiments, further, obtaining the region sequence score corresponding to the second training feature data based on the semantic segmentation prediction result corresponding to the second training feature data in the region corresponding to the image included in the second training feature data comprises:

[0204] Based on the area corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, the minimum bounding rectangle of the area corresponding to each set semantic segmentation label in the image is obtained; based on the size of the minimum bounding rectangle of the area corresponding to each set semantic segmentation label in the image, each retained semantic segmentation label is obtained; based on the size of the minimum bounding rectangle of the area corresponding to each retained semantic segmentation label in the image and the area weight calculation formula, the area weight corresponding to each retained semantic segmentation label is calculated; wherein the area weight calculation formula is preset; according to the area weight and position weight corresponding to each retained semantic segmentation label, the comprehensive weight corresponding to each retained semantic segmentation label is obtained; wherein the position weight corresponding to each retained semantic segmentation label is preset Assume that; based on the angle between the long side of the minimum circumscribed rectangle of the area corresponding to each retained semantic segmentation label in the image and the horizontal direction of the image and the center of the image corresponding to the second training feature data, obtain the image direction corresponding to the second training feature data and obtain the vertical line of the image direction corresponding to the training feature data; project the center of the area corresponding to each retained semantic segmentation label in the image onto the vertical line of the image direction corresponding to the second training feature data, and obtain the projection point coordinates corresponding to each retained semantic segmentation label; based on the projection point coordinates corresponding to each retained semantic segmentation label, the comprehensive weight and the area weight, obtain the deduction points corresponding to the second training feature data; calculate the difference between the preset full score and the deduction points corresponding to the second training feature data, and obtain the area sequence score corresponding to the second training feature data.

[0205] On the basis of the above embodiments, further, obtaining the deduction points corresponding to the second training feature data based on the projection point coordinates, comprehensive weights and regional weights corresponding to each retained semantic segmentation label includes:

[0206] Sort the projection point coordinates corresponding to each retained semantic segmentation label in order from small to large to obtain the first arrangement order of each retained semantic segmentation label; and sort the projection point coordinates corresponding to each retained semantic segmentation label in order from large to small to obtain the second arrangement order of each retained semantic segmentation label; according to the first arrangement order and the comprehensive weight and regional weight of the sorted adjacent retained semantic segmentation labels, obtain the first order deduction points corresponding to the second training feature data; and according to the second arrangement order and the comprehensive weight and regional weight of the sorted adjacent retained semantic segmentation labels, obtain the second order deduction points corresponding to the second training feature data; obtain the smaller value of the first order deduction points corresponding to the second training feature data and the second order deduction points corresponding to the training feature data as the deduction points corresponding to the training feature data.

[0207] On the basis of the above embodiments, further, the region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, obtaining the average region area and the appropriate region proportion corresponding to the second training feature data includes:

[0208] According to the area corresponding to the semantic segmentation prediction result corresponding to the second training feature data and the image included in the second training feature data, the area of ​​the area corresponding to each semantic segmentation prediction result in the image is obtained; and the number of semantic segmentation prediction results corresponding to the second training feature data is counted to obtain the total number of areas corresponding to the second training feature data; according to the area of ​​the area corresponding to each semantic segmentation prediction result in the image, the total area of ​​the area corresponding to the second training feature data is obtained; according to the total number of areas and the total area of ​​the areas corresponding to the second training feature data, the average area of ​​the area corresponding to the second training feature data is obtained; according to the area of ​​the area corresponding to each semantic segmentation prediction result in the image and the area threshold, the number of suitable areas corresponding to the second training feature data is obtained; according to the number of suitable areas corresponding to the second training feature data and the total number of areas corresponding to the second training feature data, the proportion of suitable areas corresponding to the second training feature data is obtained.

[0209] Fig.15 is a schematic diagram of the structure of a training device for a dynamic semantic segmentation model provided in the fifteenth embodiment of the present invention. Fig.15 As shown, on the basis of the above embodiments, further, the training unit 1103 includes a first calculation subunit 110309, a second calculation subunit 110310, a third calculation subunit 110311 and a second obtaining subunit 110312, wherein:

[0210] The first calculation subunit 110309 is used to calculate the first loss value corresponding to the first training feature data in the first category of training data based on the first loss function; the second calculation subunit 110310 is used to calculate the second loss value corresponding to the first training feature data in the first category of training data based on the second loss function; the third calculation subunit 110311 is used to calculate the third loss value corresponding to the first training feature data in the first category of training data based on the third loss function; the second acquisition subunit 110312 is used to obtain the loss value corresponding to each first training feature data according to the normalized loss value of each semantic segmentation prediction result of each first training feature data and the second loss function.

[0211] On the basis of the above embodiments, further, the second loss function is:

[0212]

[0213] Among them, loss c Indicates the first loss value corresponding to the first training feature data, num c It represents the number of pixels of the area corresponding to other semantic segmentation prediction results included in the area corresponding to the semantic segmentation prediction result corresponding to the first training feature data, W represents the first number of pixels corresponding to the first training feature data, and H represents the second number of pixels corresponding to the first training feature data.

[0214] On the basis of the above embodiments, further, the third loss function is:

[0215]

[0216] Among them, loss represents the third loss value corresponding to the first training feature data, loss i represents the normalized loss value corresponding to the i-th semantic segmentation prediction result corresponding to the first training feature data, I i Represents the intersection-over-union ratio corresponding to the i-th semantic segmentation prediction result.

[0217] Fig.16 is a structural schematic diagram of a semantic segmentation device for a fully mechanized coal mining face provided by the sixteenth embodiment of the present invention, such as Fig.16 As shown, the semantic segmentation device of the coal mine fully mechanized mining working face provided by the embodiment of the present invention includes a first obtaining unit 1601, a second obtaining unit 1602, a third obtaining unit 1603 and a semantic segmentation unit 1604, wherein:

[0218] The first acquisition unit 1601 is used to obtain video data based on the coal mine comprehensive mining working face and obtain the current frame video data; the second acquisition unit 1602 is used to preprocess the current frame video data and obtain the current frame image corresponding to the current frame video data; the third acquisition unit 1603 is used to perform optical flow calculation processing on the current frame image corresponding to the current frame video data and the previous frame image of the current frame image corresponding to the current frame video data to obtain the predicted feature data corresponding to the current frame video data; the semantic segmentation unit 1604 is used to obtain the semantic segmentation result corresponding to the current frame video data based on the predicted feature data corresponding to the current frame video data and the dynamic semantic segmentation model.

[0219] Specifically, when coal is mined underground using coal mining equipment, a camera will be installed to collect video data of the coal mine's comprehensive mining working face. The camera can send the video data of the coal mine's comprehensive mining working face to the first acquisition unit 1601. The first acquisition unit 1601 will receive the video data of the coal mine's comprehensive mining working face and obtain the current frame video data therefrom. The current frame video data is a frame of video data currently being processed by the server.

[0220] The second obtaining unit 1602 preprocesses the current frame video data to obtain a current frame image corresponding to the current frame video data. The preprocessing includes but is not limited to format conversion, normalization processing, etc., which are set according to actual needs and are not limited in the embodiment of the present invention.

[0221] The third obtaining unit 1603 performs optical flow calculation processing on the current frame image corresponding to the current frame video data and the previous frame image of the current frame image corresponding to the current frame video data, and can obtain the predicted feature data corresponding to the current frame video data. The predicted feature data corresponding to the current frame video data includes the current frame image and the motion information between the current frame image and the previous frame image corresponding to the current frame image.

[0222] The semantic segmentation unit 1604 inputs the predicted feature data corresponding to the current frame video data into the dynamic semantic segmentation model, performs semantic segmentation on the current frame image, and obtains the semantic segmentation result corresponding to the current frame video data. The semantic segmentation result corresponding to the current frame video data includes the semantic segmentation label corresponding to the current frame image.

[0223] The semantic segmentation device for a coal mine fully mechanized mining working face provided in an embodiment of the present invention obtains current frame video data based on video stream data of the coal mine fully mechanized mining working face; preprocesses the current frame video data to obtain a current frame image corresponding to the current frame video data; performs optical flow calculation processing on the current frame image corresponding to the current frame video data and the previous frame image of the current frame image corresponding to the current frame video data to obtain predicted feature data corresponding to the current frame video data; obtains a semantic segmentation result corresponding to the current frame video data based on the predicted feature data corresponding to the current frame video data and a dynamic semantic segmentation model, thereby improving the accuracy of dynamic semantic segmentation of the fully mechanized mining working face.

[0224] The embodiments of the apparatus provided in the embodiments of the present invention can be specifically used to execute the processing flow of the above-mentioned method embodiments, and its functions are not described in detail here, and reference can be made to the detailed description of the above-mentioned method embodiments.

[0225] Fig.17 is a schematic diagram of the physical structure of an electronic device provided by the seventeenth embodiment of the present invention, such as Fig.17 As shown, the electronic device may include: a processor 1701, a communication interface 1702, a memory 1703 and a communication bus 1704, wherein the processor 1701, the communication interface 1702 and the memory 1703 communicate with each other through the communication bus 1704. The processor 1701 may call the logic instructions in the memory 1703 to execute the method described in any of the above embodiments.

[0226] In addition, the logic instructions in the above-mentioned memory 1703 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0227] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments.

[0228] This embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program enables the computer to execute the methods provided by the above-mentioned method embodiments.

[0229] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0230] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0231] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0232] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0233] In the description of this specification, the description with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0234] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A training method for a dynamic semantic segmentation model, applied to a fully mechanized coal mining face, characterized in that: include: Preprocessing historical video data of a fully-mechanized coal mining face to obtain a video sequence of the fully-mechanized coal mining face; Performing optical flow calculation processing on the video sequence of the coal mine fully mechanized mining working face to obtain training feature data of the coal mine fully mechanized mining working face; the training feature data includes a first type of training data and a second type of training data, the first type of training data has a corresponding semantic segmentation label, and the second type of training data lacks a corresponding semantic segmentation label; wherein the semantic segmentation label is set based on the coal mine fully mechanized mining working face; A dynamic semantic segmentation model is obtained based on the first category of training data, the second category of training data and the original model; wherein, obtaining the dynamic semantic segmentation model based on the first category of training data, the second category of training data and the original model includes multiple trainings, each training includes a pre-training stage and a labeling stage; the pre-training stage performs model training based on the first category of training data to obtain an intermediate dynamic semantic segmentation model; the labeling stage performs semantic segmentation labeling on the second category of training data based on the intermediate dynamic semantic segmentation model, and screens out the second category of training data with valid semantic segmentation labels to update the first category of training data.

2. The method according to claim 1, characterized in that The performing optical flow calculation processing on the video sequence of the coal mine fully mechanized mining working face to obtain the training feature data of the coal mine fully mechanized mining working face comprises: Dividing the video sequence into frames to obtain each frame image corresponding to the video sequence; For each frame image starting from the second frame image in each frame image corresponding to the video sequence, optical flow calculation is performed based on each frame image and a previous frame image corresponding to each frame image to obtain motion information corresponding to each frame image; According to the motion information corresponding to each frame image and each frame image, the training feature data of the coal mine fully mechanized mining working face is obtained.

3. The method according to claim 1, characterized in that The performing model training based on the first type of training data to obtain the intermediate dynamic semantic segmentation model includes: Get the latest first-category training data; The model is trained according to the latest first-category training data to obtain an intermediate dynamic semantic segmentation model.

4. The method according to claim 1, characterized in that: The step of labeling the second type of training data with semantic segmentation labels based on the intermediate dynamic semantic segmentation model, and screening out the second type of training data with valid semantic segmentation labels to update the first type of training data includes: According to the second type of training data and the intermediate dynamic semantic segmentation model, obtaining a semantic segmentation prediction and recognition result corresponding to each second training feature data in the second type of training data, wherein the semantic segmentation prediction and recognition result includes a semantic segmentation prediction result; Determine, according to the semantic segmentation prediction recognition result corresponding to each second training feature data in the second type of training data and the label evaluation rule, the second training feature data with valid semantic segmentation prediction result in the second type of training data; The first category of training data is updated according to the second training feature data having valid semantic segmentation prediction results in the second category of training data to obtain updated first category of training data.

5. The method according to claim 3, characterized in that: The label evaluation rules include: Obtaining a region sequence score corresponding to the second training feature data based on a region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data; Based on the region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, obtaining an average region area and a suitable region proportion corresponding to the second training feature data; If the region sequence score, average region area and suitable region proportion corresponding to the second training feature data are all greater than their respective thresholds, the semantic segmentation prediction result corresponding to the second training feature data is valid; otherwise, the semantic segmentation prediction result corresponding to the second training feature data is invalid.

6. The method according to claim 5, characterized in that The obtaining of the region sequence score corresponding to the second training feature data based on the region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data comprises: Based on the semantic segmentation prediction result corresponding to the second training feature data, the minimum bounding rectangle of the area corresponding to each set semantic segmentation label in the image is obtained; Obtaining each retained semantic segmentation label based on the size of the minimum bounding rectangle of the area corresponding to each set semantic segmentation label in the image; Based on the size of the minimum bounding rectangle of the area corresponding to each retained semantic segmentation label in the image and the area weight calculation formula, the area weight corresponding to each retained semantic segmentation label is calculated; wherein the area weight calculation formula is preset; According to the region weight and position weight corresponding to each retained semantic segmentation label, a comprehensive weight corresponding to each retained semantic segmentation label is obtained; wherein the position weight corresponding to each retained semantic segmentation label is preset; Based on the angle between the long side of the minimum circumscribed rectangle of the area corresponding to each retained semantic segmentation label in the image and the horizontal direction of the image and the center of the image corresponding to the second training feature data, obtain the image direction corresponding to the second training feature data and obtain the vertical line of the image direction corresponding to the training feature data; Projecting the center of the region corresponding to each retained semantic segmentation label in the image onto a vertical line in the image direction corresponding to the second training feature data to obtain the coordinates of the projection point corresponding to each retained semantic segmentation label; Based on the projection point coordinates, comprehensive weights, and regional weights corresponding to each retained semantic segmentation label, a deduction score corresponding to the second training feature data is obtained; The difference between the preset full score and the deduction score corresponding to the second training feature data is calculated to obtain the regional sequence score corresponding to the second training feature data.

7. The method according to claim 6, characterized in that The step of obtaining the deduction points corresponding to the second training feature data based on the projection point coordinates, the comprehensive weights, and the regional weights corresponding to the respective retained semantic segmentation labels comprises: The projection point coordinates corresponding to each retained semantic segmentation label are sorted in order from small to large to obtain a first sorting order of each retained semantic segmentation label; and the projection point coordinates corresponding to each retained semantic segmentation label are sorted in order from large to small to obtain a second sorting order of each retained semantic segmentation label; According to the first arrangement order and the comprehensive weights and regional weights of the retained semantic segmentation labels that are adjacent to each other, a first order deduction score corresponding to the second training feature data is obtained; and according to the second arrangement order and the comprehensive weights and regional weights of the retained semantic segmentation labels that are adjacent to each other, a second order deduction score corresponding to the second training feature data is obtained; The smaller value of the first order deduction points corresponding to the second training feature data and the second order deduction points corresponding to the training feature data is obtained as the deduction points corresponding to the training feature data.

8. The method according to claim 4, characterized in that The obtaining of the average area and the proportion of the appropriate area corresponding to the second training feature data based on the area corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data comprises: According to the region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, the area of ​​the region corresponding to each semantic segmentation prediction result in the image is obtained; and the number of semantic segmentation prediction results corresponding to the second training feature data is counted to obtain the total number of regions corresponding to the second training feature data; Obtaining the total area of ​​the region corresponding to the second training feature data according to the area of ​​the region corresponding to each semantic segmentation prediction result in the image; Obtaining an average area of ​​regions corresponding to the second training feature data according to the total number of regions and the total area of ​​regions corresponding to the second training feature data; Obtaining the number of suitable regions corresponding to the second training feature data according to the area of ​​the region corresponding to each semantic segmentation prediction result in the image and the area threshold; The proportion of suitable areas corresponding to the second training feature data is obtained according to the number of suitable areas corresponding to the second training feature data and the total number of areas corresponding to the second training feature data.

9. The method according to any one of claims 1 to 8, characterized in that: The performing model training based on the first type of training data to obtain the intermediate dynamic semantic segmentation model includes: Calculate a first loss value corresponding to the first training feature data in the first type of training data based on a first loss function; Calculate a second loss value corresponding to the first training feature data in the first type of training data based on a second loss function; Calculate a third loss value corresponding to the first training feature data in the first type of training data based on a third loss function; The first loss value, the second loss value, and the third loss value are summed to obtain a total loss value corresponding to the first training feature data.

10. The method according to claim 9, characterized in that The second loss function is: Among them, loss c Indicates the first loss value corresponding to the first training feature data, num c It represents the number of pixels of the area corresponding to other semantic segmentation prediction results included in the area corresponding to the semantic segmentation prediction result corresponding to the first training feature data, W represents the first number of pixels corresponding to the first training feature data, and H represents the second number of pixels corresponding to the first training feature data.

11. The method according to claim 9, characterized in that The third loss function is: Among them, loss represents the third loss value corresponding to the first training feature data, loss i represents the normalized loss value corresponding to the i-th semantic segmentation prediction result corresponding to the first training feature data, I i Represents the intersection-over-union ratio corresponding to the i-th semantic segmentation prediction result.

12. A semantic segmentation method for a fully mechanized coal mining face, characterized in that: The dynamic semantic segmentation model according to any one of claims 1 to 11 is adopted, comprising: Based on the video data of the fully mechanized mining face of the coal mine, the current frame video data is obtained; Preprocessing the current frame video data to obtain a current frame image corresponding to the current frame video data; Performing optical flow calculation processing on a current frame image corresponding to the current frame video data and a previous frame image of the current frame image corresponding to the current frame video data to obtain predicted feature data corresponding to the current frame video data; According to the predicted feature data corresponding to the current frame video data and the dynamic semantic segmentation model, a semantic segmentation result corresponding to the current frame video data is obtained.

13. A training device for a dynamic semantic segmentation model, applied to a fully mechanized coal mining face, characterized in that: include: A preprocessing unit, used for preprocessing historical video data of a fully-mechanized coal mining working face to obtain a video sequence of the fully-mechanized coal mining working face; An optical flow calculation processing unit is used to perform optical flow calculation processing on the video sequence of the coal mine fully mechanized mining working face to obtain training feature data of the coal mine fully mechanized mining working face; the training feature data includes a first type of training data and a second type of training data, the first type of training data has a corresponding semantic segmentation label, and the second type of training data lacks a corresponding semantic segmentation label; wherein the semantic segmentation label is set based on the coal mine fully mechanized mining working face; A training unit is used to obtain a dynamic semantic segmentation model based on the first category of training data, the second category of training data and the original model; wherein, obtaining the dynamic semantic segmentation model based on the first category of training data, the second category of training data and the original model includes multiple trainings, and each training includes a pre-training stage and a labeling stage; the pre-training stage performs model training based on the first category of training data to obtain an intermediate dynamic semantic segmentation model; the labeling stage performs semantic segmentation labeling on the second category of training data based on the intermediate dynamic semantic segmentation model, and screens out the second category of training data with valid semantic segmentation labels to update the first category of training data.

14. The device according to claim 13, characterized in that The training unit comprises: A frame division subunit, used for dividing the video sequence into frames to obtain each frame image corresponding to the video sequence; an optical flow calculation subunit, configured to perform optical flow calculation on each frame image starting from the second frame image in each frame image corresponding to the video sequence based on each frame image and a previous frame image corresponding to each frame image, so as to obtain motion information corresponding to each frame image; The acquisition subunit is used to obtain the training feature data of the coal mine fully mechanized mining working face according to the motion information corresponding to each frame image and each frame image.

15. The device according to claim 13, characterized in that The training unit comprises: An acquisition subunit, used to acquire the latest first-category training data; The training subunit is used to perform model training according to the latest first-category training data to obtain an intermediate dynamic semantic segmentation model.

16. The device according to claim 13, characterized in that The training unit comprises: A prediction subunit, configured to obtain, according to the second type of training data and the intermediate dynamic semantic segmentation model, a semantic segmentation prediction and recognition result corresponding to each second training feature data in the second type of training data, wherein the semantic segmentation prediction and recognition result includes a semantic segmentation prediction result; A determination subunit, configured to determine, according to the semantic segmentation prediction recognition result corresponding to each second training feature data in the second type of training data and a label evaluation rule, second training feature data with valid semantic segmentation prediction results in the second type of training data; An updating subunit is used to update the first category of training data according to the second category of training data having valid semantic segmentation prediction results, so as to obtain updated first category of training data.

17. The device according to claim 16, characterized in that The label evaluation rules include: Obtaining a region sequence score corresponding to the second training feature data based on a region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data; Based on the region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, obtaining an average region area and a suitable region proportion corresponding to the second training feature data; If the region sequence score, average region area and suitable region proportion corresponding to the second training feature data are all greater than their respective thresholds, the semantic segmentation prediction result corresponding to the second training feature data is valid; otherwise, the semantic segmentation prediction result corresponding to the second training feature data is invalid.

18. The device according to claim 17, characterized in that The obtaining, based on the semantic segmentation prediction result corresponding to the second training feature data, of the region corresponding to the image included in the second training feature data, the region sequence score corresponding to the second training feature data comprises: Based on the semantic segmentation prediction result corresponding to the second training feature data, the minimum bounding rectangle of the area corresponding to each set semantic segmentation label in the image is obtained; Obtaining each retained semantic segmentation label based on the size of the minimum bounding rectangle of the area corresponding to each set semantic segmentation label in the image; Based on the size of the minimum bounding rectangle of the area corresponding to each retained semantic segmentation label in the image and the area weight calculation formula, the area weight corresponding to each retained semantic segmentation label is calculated; wherein the area weight calculation formula is preset; According to the region weight and position weight corresponding to each retained semantic segmentation label, a comprehensive weight corresponding to each retained semantic segmentation label is obtained; wherein the position weight corresponding to each retained semantic segmentation label is preset; Based on the angle between the long side of the minimum circumscribed rectangle of the area corresponding to each retained semantic segmentation label in the image and the horizontal direction of the image and the center of the image corresponding to the second training feature data, obtain the image direction corresponding to the second training feature data and obtain the vertical line of the image direction corresponding to the training feature data; Projecting the center of the region corresponding to each retained semantic segmentation label in the image onto a vertical line in the image direction corresponding to the second training feature data to obtain the coordinates of the projection point corresponding to each retained semantic segmentation label; Based on the projection point coordinates, comprehensive weights, and regional weights corresponding to each retained semantic segmentation label, a deduction score corresponding to the second training feature data is obtained; The difference between the preset full score and the deduction score corresponding to the second training feature data is calculated to obtain the regional sequence score corresponding to the second training feature data.

19. The device according to claim 18, characterized in that The step of obtaining the deduction points corresponding to the second training feature data based on the projection point coordinates, the comprehensive weights, and the regional weights corresponding to the respective retained semantic segmentation labels comprises: The projection point coordinates corresponding to each retained semantic segmentation label are sorted in order from small to large to obtain a first sorting order of each retained semantic segmentation label; and the projection point coordinates corresponding to each retained semantic segmentation label are sorted in order from large to small to obtain a second sorting order of each retained semantic segmentation label; According to the first arrangement order and the comprehensive weights and regional weights of the retained semantic segmentation labels that are adjacent to each other, a first order deduction score corresponding to the second training feature data is obtained; and according to the second arrangement order and the comprehensive weights and regional weights of the retained semantic segmentation labels that are adjacent to each other, a second order deduction score corresponding to the second training feature data is obtained; The smaller value of the first order deduction points corresponding to the second training feature data and the second order deduction points corresponding to the training feature data is obtained as the deduction points corresponding to the training feature data.

20. The device according to claim 17, characterized in that The obtaining of the average area and the proportion of the appropriate area corresponding to the second training feature data based on the area corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data comprises: According to the region corresponding to the semantic segmentation prediction result corresponding to the second training feature data in the image included in the second training feature data, the area of ​​the region corresponding to each semantic segmentation prediction result in the image is obtained; and the number of semantic segmentation prediction results corresponding to the second training feature data is counted to obtain the total number of regions corresponding to the second training feature data; Obtaining the total area of ​​the region corresponding to the second training feature data according to the area of ​​the region corresponding to each semantic segmentation prediction result in the image; Obtaining an average area of ​​regions corresponding to the second training feature data according to the total number of regions and the total area of ​​regions corresponding to the second training feature data; Obtaining the number of suitable regions corresponding to the second training feature data according to the area of ​​the region corresponding to each semantic segmentation prediction result in the image and the area threshold; The proportion of suitable areas corresponding to the second training feature data is obtained according to the number of suitable areas corresponding to the second training feature data and the total number of areas corresponding to the second training feature data.

21. The device according to any one of claims 13 to 20, characterized in that The training unit comprises: A first calculation subunit, configured to calculate a first loss value corresponding to first training feature data in the first type of training data based on a first loss function; A second calculation subunit, configured to calculate a second loss value corresponding to the first training feature data in the first type of training data based on a second loss function; A third calculation subunit, configured to calculate a third loss value corresponding to the first training feature data in the first type of training data based on a third loss function; The second obtaining subunit is used to sum the first loss value, the second loss value and the third loss value to obtain a total loss value corresponding to the first training feature data.

22. The device according to claim 21, characterized in that The second loss function is: Among them, loss c Indicates the first loss value corresponding to the first training feature data, num c It represents the number of pixels of the area corresponding to other semantic segmentation prediction results included in the area corresponding to the semantic segmentation prediction result corresponding to the first training feature data, W represents the first number of pixels corresponding to the first training feature data, and H represents the second number of pixels corresponding to the first training feature data.

23. The device according to claim 21, characterized in that The second loss function is: Among them, loss represents the third loss value corresponding to the first training feature data, loss i represents the normalized loss value corresponding to the i-th semantic segmentation prediction result corresponding to the first training feature data, I i Represents the intersection-over-union ratio corresponding to the i-th semantic segmentation prediction result.

24. A semantic segmentation device for a fully mechanized coal mining face, characterized in that: include: The first acquisition unit is used to obtain video data based on the fully mechanized mining working face of the coal mine and obtain current frame video data; A second obtaining unit, configured to preprocess the current frame video data to obtain a current frame image corresponding to the current frame video data; A third obtaining unit is used to perform optical flow calculation processing on a current frame image corresponding to the current frame video data and a previous frame image of the current frame image corresponding to the current frame video data to obtain prediction feature data corresponding to the current frame video data; The semantic segmentation unit is used to obtain the semantic segmentation result corresponding to the current frame video data according to the predicted feature data corresponding to the current frame video data and the dynamic semantic segmentation model.

25. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program / instruction, which implements the steps of the method according to any one of claims 1 to 12 when executed by a processor.

27. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images