Method for snapshot vehicle cleaning based on depth features in video and related device

Abstract

The invention discloses a method for cleaning snapshot vehicles in a video based on depth features and a related device. The method comprises the following steps: determining a target detection network and a classifier constructed based on deep learning; receiving video data collected from the vehicle cleaning area; inputting the original image data into a target detection network to detect image data of a region where the vehicle is located; in the classifier, target image features are extracted from the regional image data, and a first probability of vehicle cleaning is calculated according to the target image features; in a representative dimension, the first probability is corrected according to the target image features, and a second probability of vehicle cleaning is obtained; and screening out part of the original image data according to the second probability to serve as target image data representing vehicle cleaning. According to the embodiment of the invention, the representativeness of the selected target image data can be effectively improved, the information amount of the cleaned vehicle in the target image data is ensured, the video data browsing frequency of supervision personnel is reduced, and the vehicle cleaning supervision efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a method for capturing vehicle cleaning based on depth features in a video and a related device. Background Art

[0002] The personnel and goods in the park rely on vehicle transportation. According to factors such as the frequency of use of the park's vehicles and the parking environment, a reasonable cleaning plan is formulated to clean the vehicles.

[0003] For example, frequently used park service vehicles may need to be cleaned once a week; while for spare vehicles that are parked for a long time, the cleaning cycle can be appropriately extended.

[0004] Under the trend of intelligent management of the park, cameras are installed in the areas where vehicles are cleaned in the park. The cameras continuously collect video data, use target detection algorithms to detect whether there are vehicles in the video data, capture video clips with vehicles, and sample multiple frames of image data from the video clips at a fixed frequency as representative images of the vehicle cleaning process. This makes it easier for managers to supervise the vehicle cleaning process. When the representative image lacks operational information about vehicle cleaning, managers will browse the video clips to supervise whether the vehicle cleaning process complies with regulations.

[0005] However, the cleaning procedures for different vehicles are different, and the operations of different operators are also different, resulting in many variations in the vehicle cleaning process. Fixed capture of representative images will result in a lot of missing operational information on vehicle cleaning in the representative images, causing supervisors to frequently browse video clips, reducing the efficiency of supervising vehicle cleaning. Summary of the invention

[0006] In view of this, the present invention provides a method and related device for capturing vehicle cleaning in a video based on depth features, so as to increase the amount of information of operation information captured in a representative image of the vehicle cleaning.

[0007] A first aspect of the present invention provides a method for capturing vehicle cleaning in a video based on depth features, comprising:

[0008] Identify target detection networks and classifiers built on deep learning;

[0009] Receiving video data collected from a vehicle washing area; the video data includes multiple frames of original image data;

[0010] Inputting the original image data into the target detection network to detect the image data of the area where the vehicle is located;

[0011] In the classifier, extracting target image features from the regional image data and calculating a first probability of washing the vehicle based on the target image features;

[0012] In a representative dimension, the first probability is modified according to the target image feature to obtain a second probability of washing the vehicle;

[0013] Part of the original image data is screened out according to the second probability as target image data representing the vehicle being cleaned.

[0014] A second aspect of the present invention provides a device for capturing vehicle cleaning in a video based on depth features, comprising:

[0015] Model determination module, used to determine the target detection network and the classifier built based on deep learning;

[0016] A video data receiving module, used to receive video data collected from a vehicle washing area; the video data includes multiple frames of original image data;

[0017] A regional image data detection module, used for inputting the original image data into the target detection network to detect regional image data where the vehicle is located;

[0018] A first probability calculation module, used in the classifier to extract target image features from the regional image data and calculate a first probability of cleaning the vehicle based on the target image features;

[0019] A second probability calculation module is used to correct the first probability according to the target image feature in a representative dimension to obtain a second probability of cleaning the vehicle;

[0020] The target image screening module is used to screen out part of the original image data according to the second probability as target image data representing the vehicle being cleaned.

[0021] A third aspect of the present invention provides an electronic device, the electronic device comprising:

[0022] at least one processor; and

[0023] a memory communicatively connected to the at least one processor; wherein,

[0024] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for capturing vehicle cleaning in a video based on depth features as described in the first aspect above.

[0025] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for capturing vehicle cleaning in a video based on depth features as described in the first aspect above.

[0026] A fifth aspect of the present invention provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the method for capturing vehicle cleaning in a video based on depth features as described in the first aspect above.

[0027] In this embodiment, a target detection network and a classifier built based on deep learning are determined; video data collected from a vehicle cleaning area is received; the video data has multiple frames of original image data; the original image data is input into the target detection network to detect the image data of the area where the vehicle is located; in the classifier, the target image features are extracted from the area image data and the first probability of cleaning the vehicle is calculated based on the target image features; in the representative dimension, the first probability is corrected based on the target image features to obtain the second probability of cleaning the vehicle; based on the second probability, part of the original image data is screened out as the target image data representing the cleaning of the vehicle. This embodiment uses a cascaded solution to select target image data representing the cleaning of the vehicle. The first level detects the area where the vehicle is located to reduce background interference. The second level classifies the area where the vehicle is located for cleaning vehicles. The classification is corrected to the representative dimension based on the characteristics of the vehicle, which can effectively improve the representativeness of the selected target image data, ensure the amount of information about the cleaning of the vehicle in the target image data, reduce the frequency of supervisors browsing video data, and improve the efficiency of supervising the cleaning of vehicles.

[0028] In addition, the target detection network can reuse previous projects, effectively reducing the development workload. The binary classification task of the classifier is relatively simple and the structure is lightweight. Moreover, both have a small resource burden and are suitable for deployment on edge computing nodes with limited resources.

[0029] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1It is a flowchart of a method for capturing vehicle cleaning in a video based on depth features provided in Example 1 of the present invention.

[0032] Figure 2 It is a structural diagram of a classifier provided in Example 1 of the present invention.

[0033] Figure 3 It is a structural schematic diagram of a device for capturing vehicle cleaning in a video based on depth features provided in Embodiment 2 of the present invention.

[0034] Figure 4 It is a structural schematic diagram of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can cover sequential implementations other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Embodiment 1

[0038] See also Figure 1 , shows a flow chart of a method for capturing vehicle cleaning based on depth features in a video provided by a first embodiment of the present invention. The method can be performed by a device for capturing vehicle cleaning based on depth features in a video. The device for capturing vehicle cleaning based on depth features in a video can be implemented in the form of hardware and / or software. The device for capturing vehicle cleaning based on depth features in a video can be configured in an electronic device. Figure 1 As shown, the method includes:

[0039] Step 101: Determine a target detection network and a classifier built based on deep learning.

[0040] In this embodiment, a target detection network and a classifier can be constructed based on deep learning.

[0041] Among them, the target detection network can be divided into one-stage and two-stage.

[0042] Two-stage belongs to segment to segment, which means that the target detection operation is completed in two steps. The first step is to use various convolutional neural networks as the backbone of the target detection network to extract features from the original image data, perform coarse classification (distinguishing foreground and background) and coarse positioning (anchor) based on the features, and obtain candidate areas. The second step is to classify the candidate areas (i.e. vehicles) in the classification network of the target detection network.

[0043] Exemplarily, the two-stage target detection operation may include R-CNN (Region-CNN, region-based convolutional neural network), Fast R-CNN (fast regional convolutional neural network), Faster R-CNN (faster regional convolutional neural network), R-FCN (Region-based fully convolutional network, region-based fully convolutional network), and the like.

[0044] One-stage is end-to-end, which means that the target detection operation is completed in one step. It does not search for candidate areas separately. The original image data is input into an overall network, and the generated detection results also include the vehicle's location and category information.

[0045] Exemplarily, the one-stage target detection operation may include SSD (Single Shot Multibox Detector), YOLO (You Only Look Once), and the like.

[0046] Generally speaking, two-stage detection has higher accuracy but slightly slower detection speed, while one-stage detection has faster speed but lower accuracy. In this embodiment, one-stage or two-stage can be selected based on factors such as the resources of the edge computing node and the real-time requirements of the detection. This embodiment does not impose any restrictions on this.

[0047] In some cases, if a target detection network has been built in advance to detect the presence of a vehicle, the target detection network can be reused directly, which can avoid a large amount of repeated development activities and effectively reduce development costs.

[0048] For the target detection network and classifier, image data with and / or without vehicles can be collected, and the detection frame of the vehicle and the category of whether there is an operator cleaning the vehicle can be marked on the image data to construct a data set.

[0049] The target detection network is supervisedly trained using the image data in the dataset as samples and the detection box as the label Tag, and the target detection network is verified based on the dataset, so that the target detection network can be used to detect whether there is a vehicle.

[0050] The classifier is trained in a supervised manner using the image data in the dataset as samples and the categories as tags, and is verified based on the dataset so that the classifier can be used to identify whether there are operators cleaning the vehicle.

[0051] Deploy edge computing nodes in the park, and deploy target detection networks and classifiers in edge computing nodes.

[0052] Step 102: Receive video data collected from the vehicle washing area.

[0053] In this embodiment, one or more cameras at different angles can be set up in the area where vehicles are washed in the park, and each camera is connected to an edge computing node in the park. At this time, the edge computing node can receive various video data collected by each camera in real time, and capture representative images of the washed vehicles from each video data. In this way, both the real-time detection and the privacy of the video data are guaranteed.

[0054] Of course, the video data collected by the camera can also be transmitted to the cloud through the mobile network, and the target detection network and classifier are deployed in the cloud to capture representative images of the cleaning vehicle from the video data of each channel. This embodiment does not limit this.

[0055] Step 103: Input the original image data into the target detection network to detect the image data of the area where the vehicle is located.

[0056] In this embodiment, the video data contains multiple frames of original image data, and part of the original image data can be selected by frame extraction, and the selected original image data are traversed, and the current original image data is input into the target detection network to detect the image data of the area where the vehicle is located.

[0057] Step 104: In the classifier, extract target image features from the regional image data and calculate a first probability of washing the vehicle according to the target image features.

[0058] In this embodiment, the regional image data is scaled to a uniform size. If the scaling is completed, the regional image data is input into a classifier, and the classifier performs a binary classification operation on the regional image data. During the binary classification process, target image features are extracted from the regional image data, and a first probability of cleaning the vehicle is calculated based on the target image features.

[0059] In the specific implementation, Figure 2 As shown, the classifier includes a backbone structure Backbone and a classification head Head.

[0060] Among them, the backbone structure Backbone includes single structures such as convolutional layers, pooling layers, activation functions (such as ReLU (Rectified Linear Unit, rectified linear unit)), etc., or its composite structure (such as residual structure, ASPP (Atrous spatial pyramid pooling, hollow pyramid pooling), etc.). The regional image data can be input into the backbone structure Backbone to extract features and recorded as target image features.

[0061] In addition, the classification head includes structures such as a fully connected layer and an activation function (such as Sigmoid, etc.), and the target image features can be input into the classification head to calculate the first probability of cleaning the vehicle.

[0062] Of course, the above-mentioned classifier structure is only used as an example. When implementing this embodiment, other classifier structures can be set according to actual conditions, and this embodiment does not limit this. In addition, in addition to the above-mentioned classifier structure, those skilled in the art can also adopt other classifier structures according to actual needs, and this embodiment does not limit this.

[0063] Step 105: In a representative dimension, the first probability is corrected according to the target image feature to obtain a second probability of cleaning the vehicle.

[0064] In practical applications, the first probability may be modified on the basis of the target image features in a representative dimension (ie, whether it has the ability to serve as a representative image of the vehicle cleaning process) to obtain a second probability of cleaning the vehicle.

[0065] In one embodiment of the present invention, step 105 may include the following steps:

[0066] Step 1051: Query representative sample image data in a preset database.

[0067] In this embodiment, representative sample image data can be set for different types of vehicles from video clips recorded historically during vehicle washing, that is, the sample image data has the ability to serve as a representative image of the vehicle washing process, and the content of the sample image data is vehicle washing.

[0068] For example, the front of a vehicle includes the front face, headlights and bumper, and the front of the vehicle is one of the parts that are easily contaminated with dust, mud and other dirt. Therefore, when the user uses tools to clean key parts of the front face, headlights and bumper of the vehicle, sample image data can be sampled and set.

[0069] Furthermore, when cleaning the front face, attention should be paid to cleaning the dust and debris inside the air intake grille, because these deposits may affect the engine's air intake efficiency. At this time, when the user uses tools to clean the dust and debris on the air intake grille, sample image data can be sampled and set.

[0070] When cleaning the headlights, use a mild detergent and a soft cloth to wipe them to avoid scratching the lampshade. At the same time, make sure that the gaps around the lampshade are cleaned to prevent long-term accumulation of dirt and corrosion. At this time, when the user uses the detergent box cloth to wipe the headlights, sample image data can be set.

[0071] When cleaning the bumper, there are usually many stains and scratches on the bumper. When cleaning, pay attention to check whether there is any damage, and focus on cleaning the bottom, because this position is easily stained with stubborn stains such as asphalt and oil stains on the road. At this time, when the user uses the tool to clean the bottom of the bumper, sample image data can be set.

[0072] The sample image data is stored in the database with the type of vehicle as an index. Then, when a representative image of a cleaned vehicle is captured, the corresponding sample image data can be read from the database according to the type of the current vehicle output by the target detection network.

[0073] Since the correlation between the classifier and the target detection network is weak, the database can be expanded and the classifier can be fine-tuned according to the actual vehicle situation, which is highly flexible.

[0074] Step 1052: In the classifier, extract sample image features from the sample image data.

[0075] In this embodiment, the sample image data is scaled to a uniform size. If the scaling is completed, the sample image data is input into a classifier, and the classifier performs a binary classification operation on the sample image data. During the binary classification process, sample image features are extracted from the sample image data.

[0076] Generally speaking, the operation of extracting sample image features from sample image data using a classifier can be completed offline, and the sample image features of the sample image data are stored together with the sample image data in a database. Then, when a representative image of a cleaned vehicle is captured, the corresponding sample image features can be read from the database based on the type of the current vehicle output by the target detection network.

[0077] In the specific implementation, the classifier includes a backbone structure Backbone and a classification head Head.

[0078] Then, the sample image data can be input into the backbone structure Backbone to extract features, which are recorded as sample image features. At this time, the sample image features are intercepted and input into the classification head Head, and there is no need to calculate the probability of cleaning the vehicle.

[0079] Of course, the above-mentioned classifier structure is only used as an example. When implementing this embodiment, other classifier structures can be set according to actual conditions, and this embodiment does not limit this. In addition, in addition to the above-mentioned classifier structure, those skilled in the art can also adopt other classifier structures according to actual needs, and this embodiment does not limit this.

[0080] Step 1053: Calculate the similarity between the target image features and the sample image features.

[0081] In this embodiment, the similarity between the target image feature of the regional image data and the sample image feature of each frame of sample image data may be calculated by using methods such as cosine angle.

[0082] Step 1054: modify the first probability according to the similarity to obtain a second probability of cleaning the vehicle.

[0083] In this embodiment, the similarity between the target image features of the regional image data and the sample image features of the sample image data reflects the representativeness of the regional image data to a certain extent. Therefore, for the given sample image data, the first probability can be corrected using the corresponding similarity to obtain the second probability of cleaning the vehicle.

[0084] The number of the second probabilities is equal to the number of sample image data.

[0085] For example, if the similarity is greater than or equal to a preset first threshold, reflecting that the regional image data is highly representative, the similarity can be added to a preset first value to obtain a first weight, wherein the first value is a default empirical value, which can make the first weight greater than 1, thereby amplifying the first probability.

[0086] At this time, the product of the first probability and the first weight is calculated as the second probability of washing the vehicle.

[0087] If the similarity is less than the preset first threshold, reflecting that the regional image data is less representative, the similarity can be subtracted from the preset second value to obtain a second weight, wherein the second value is a default empirical value, which can make the second weight greater than 0 and less than 1, thereby reducing the first probability.

[0088] At this time, the product of the first probability and the second weight is calculated as the second probability of washing the vehicle.

[0089] Of course, the above-mentioned method of correcting the first probability based on the similarity is only used as an example. When implementing this embodiment, other methods of correcting the first probability based on the similarity can be set according to actual conditions, and this embodiment does not limit this. In addition, in addition to the above-mentioned method of correcting the first probability based on the similarity, those skilled in the art can also adopt other methods of correcting the first probability based on the similarity according to actual needs, and this embodiment does not limit this.

[0090] Step 106: Filter out part of the original image data according to the second probability as target image data representing the cleaned vehicle.

[0091] In practical applications, multiple second probabilities corresponding to each frame of original image data may be traversed, and part of the original image data may be screened out according to the second probabilities as target image data (ie, representative image) representing the vehicle being cleaned.

[0092] In one embodiment of the present invention, step 106 may include the following steps:

[0093] Step 1061 : For each frame of original image data, if the second probability with the highest value is greater than a preset second threshold, the original image data is marked as reference image data.

[0094] In practical applications, a frame of original image data has multiple second probabilities. At this time, for a given original image data, the second probability with the largest value can be screened out from the multiple second probabilities as the final second probability of the original image data.

[0095] If the final second probability is greater than the preset second threshold, the original image data can be marked as reference image data. At this time, it can be considered that the reference image data matches the sample image data corresponding to the second probability with the largest value, and the next step of representative image screening is performed.

[0096] If the final second probability is less than or equal to the preset second threshold, the original image data is ignored.

[0097] Step 1062: Eliminate abnormal values ​​from all reference image data in the video data.

[0098] In this embodiment, since the contents of adjacent original image data are similar, in order to avoid duplication, the reference image data can be used as data points, and all the reference image data in the video data can be processed using the exception processing algorithm in the preprocessing to eliminate abnormal values, that is, to eliminate certain abnormal reference image data.

[0099] In a specific implementation, a slidable window is added to all reference image data in the video data.

[0100] If the reference image data located at non-midpoints of the window all match the sample image data of the same frame, and the sample image data matched by the reference image data located at the midpoint of the window is different from the sample image data matched by the reference image data located at the non-midpoints of the window, then the reference image data located at the midpoint of the window will be eliminated as an outlier.

[0101] Step 1063: If the outliers are eliminated, the video data is divided into multiple segments.

[0102] In this embodiment, if outliers are removed from the video data, the video data may be divided into a plurality of segments, wherein the reference image data in the same segment all match the same frame of sample image data.

[0103] Step 1064 : In each segment, select reference image data corresponding to the second probability with the highest value as target image data.

[0104] In each segment, the final second probabilities of the reference image data may be compared, and the reference image data corresponding to the final second probability with the highest value may be screened out as the target image data.

[0105] In this embodiment, a target detection network and a classifier built based on deep learning are determined; video data collected from a vehicle cleaning area is received; the video data has multiple frames of original image data; the original image data is input into the target detection network to detect the image data of the area where the vehicle is located; in the classifier, the target image features are extracted from the area image data and the first probability of cleaning the vehicle is calculated based on the target image features; in the representative dimension, the first probability is corrected based on the target image features to obtain the second probability of cleaning the vehicle; based on the second probability, part of the original image data is screened out as the target image data representing the cleaning of the vehicle. This embodiment uses a cascaded solution to select target image data representing the cleaning of the vehicle. The first level detects the area where the vehicle is located to reduce background interference. The second level classifies the area where the vehicle is located for cleaning vehicles. The classification is corrected to the representative dimension based on the characteristics of the vehicle, which can effectively improve the representativeness of the selected target image data, ensure the amount of information about the cleaning of the vehicle in the target image data, reduce the frequency of supervisors browsing video data, and improve the efficiency of supervising the cleaning of vehicles.

[0106] In addition, the target detection network can reuse previous projects, effectively reducing the development workload. The binary classification task of the classifier is relatively simple and the structure is lightweight. Moreover, both have a small resource burden and are suitable for deployment on edge computing nodes with limited resources.

[0107] Embodiment 2

[0108] See also Figure 3 , shows a schematic diagram of the structure of a device for capturing vehicle cleaning based on depth features in a video provided by Embodiment 2 of the present invention. Figure 3 As shown, the device comprises:

[0109] A model determination module 301 is used to determine a target detection network and a classifier built based on deep learning;

[0110] The video data receiving module 302 is used to receive the video data collected from the vehicle washing area; the video data includes multiple frames of original image data;

[0111] The regional image data detection module 303 is used to input the original image data into the target detection network to detect the regional image data where the vehicle is located;

[0112] A first probability calculation module 304 is used to extract target image features from the regional image data in the classifier and calculate a first probability of cleaning the vehicle based on the target image features;

[0113] A second probability calculation module 305 is used to correct the first probability according to the target image feature in a representative dimension to obtain a second probability of cleaning the vehicle;

[0114] The target image screening module 306 is used to screen out part of the original image data according to the second probability as target image data representing the vehicle being cleaned.

[0115] In one embodiment of the present invention, the classifier includes a backbone structure and a classification head;

[0116] The first probability calculation module 304 includes:

[0117] A target image feature extraction module, used for inputting the regional image data into the backbone structure to extract target image features;

[0118] The target image feature classification module is used to input the target image feature into the classification head to calculate the first probability of cleaning the vehicle.

[0119] In one embodiment of the present invention, the second probability calculation module 305 includes:

[0120] A sample image data query module is used to query representative sample image data in a preset database; the content of the sample image data is a cleaning vehicle;

[0121] A sample image feature extraction module, used for extracting sample image features from the sample image data in the classifier;

[0122] A similarity calculation module, used to calculate the similarity between the target image features and the sample image features;

[0123] The probability correction module is used to correct the first probability according to the similarity to obtain a second probability of cleaning the vehicle.

[0124] In one embodiment of the present invention, the classifier includes a backbone structure and a classification head;

[0125] The sample image feature extraction module comprises:

[0126] A backbone extraction module, used for inputting the sample image data into the backbone structure to extract sample image features;

[0127] The header interception module is used to intercept the input of the sample image features into the classification header.

[0128] In one embodiment of the present invention, the probability correction module includes:

[0129] A first weight calculation module, configured to add the similarity to a preset first value to obtain a first weight if the similarity is greater than or equal to a preset first threshold; the first weight is greater than 1;

[0130] A first product calculation module, used for calculating the product between the first probability and the first weight as a second probability of cleaning the vehicle;

[0131] A second weight calculation module, configured to, if the similarity is less than a preset first threshold, subtract the similarity from a preset second value to obtain a second weight; the second weight is greater than 0 and less than 1;

[0132] The second product calculation module is used to calculate the product of the first probability and the second weight as the second probability of cleaning the vehicle.

[0133] In one embodiment of the present invention, the target image screening module 306 includes:

[0134] A reference image data marking module, for marking the original image data of each frame as reference image data if the second probability with the highest value is greater than a preset second threshold;

[0135] An outlier elimination module, used for eliminating outliers from all the reference image data in the video data;

[0136] A segmentation module is used to segment the video data into multiple segments if the outliers are eliminated; the reference image data in the same segment all match the sample image data of the same frame;

[0137] The segment screening module is used to screen out the original image data corresponding to the second probability with the highest value in each of the segments as the target image data.

[0138] In one embodiment of the present invention, the outlier removal module includes:

[0139] A window adding module, used for adding a slidable window to all the reference image data in the video data;

[0140] A window elimination module is used to eliminate the reference image data located at the midpoint of the window as an outlier if the reference image data located at non-midpoints of the window all match the sample image data of the same frame, and the sample image data matched by the reference image data located at the midpoint of the window is different from the sample image data matched by the reference image data located at non-midpoints of the window.

[0141] The device for capturing vehicle cleaning in a video based on depth features provided in an embodiment of the present invention can execute the method for capturing vehicle cleaning in a video based on depth features provided in any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method for capturing vehicle cleaning in a video based on depth features.

[0142] Embodiment 3

[0143] See also Figure 4 , showing a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0144] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0145] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0146] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the method of capturing vehicle cleaning based on depth features in the video.

[0147] In some embodiments, the method for capturing vehicle cleaning based on deep features in a video may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for capturing vehicle cleaning based on deep features in the video described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the method for capturing vehicle cleaning based on deep features in a video by any other appropriate means (e.g., by means of firmware).

[0148] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0149] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0150] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0151] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0152] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0153] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0154] Embodiment 4

[0155] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the method for capturing vehicle cleaning in a video based on depth features as provided in any embodiment of the present invention.

[0156] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0157] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0158] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for capturing vehicle cleaning in a video based on depth features, characterized in that: include: Identify target detection networks and classifiers built on deep learning; Receiving video data collected from a vehicle washing area; The video data contains multiple frames of original image data; Inputting the original image data into the target detection network to detect the image data of the area where the vehicle is located; In the classifier, extracting target image features from the regional image data and calculating a first probability of washing the vehicle based on the target image features; In a representative dimension, the first probability is modified according to the target image feature to obtain a second probability of washing the vehicle; Part of the original image data is screened out according to the second probability as target image data representing the vehicle being cleaned.

2. The method according to claim 1, characterized in that The classifier includes a backbone structure and a classification head; In the classifier, extracting target image features from the regional image data and calculating a first probability of cleaning the vehicle according to the target image features include: Inputting the regional image data into the backbone structure to extract target image features; The target image features are input into the classification head to calculate a first probability of cleaning the vehicle.

3. The method according to claim 1 or 2, characterized in that: The first probability is corrected according to the target image feature in the representative dimension to obtain a second probability of washing the vehicle, including: Querying a preset database for representative sample image data; the sample image data is about cleaning vehicles; In the classifier, extracting sample image features from the sample image data; Calculating the similarity between the target image features and the sample image features; The first probability is modified according to the similarity to obtain a second probability of cleaning the vehicle.

4. The method according to claim 3, characterized in that: The classifier includes a backbone structure and a classification head; The step of extracting sample image features from the sample image data in the classifier includes: Inputting the sample image data into the backbone structure to extract sample image features; The sample image features are intercepted and input into the classification head.

5. The method according to claim 3, characterized in that: The step of correcting the first probability according to the similarity to obtain a second probability of cleaning the vehicle includes: If the similarity is greater than or equal to a preset first threshold, the similarity is added to a preset first value to obtain a first weight; the first weight is greater than 1; Calculating the product of the first probability and the first weight as a second probability of cleaning the vehicle; If the similarity is less than a preset first threshold, subtract the similarity from a preset second value to obtain a second weight; the second weight is greater than 0 and less than 1; The product of the first probability and the second weight is calculated as the second probability of washing the vehicle.

6. The method according to claim 3, characterized in that The step of selecting a portion of the original image data according to the second probability as target image data representing the vehicle to be cleaned includes: For the original image data of each frame, if the second probability with the highest value is greater than a preset second threshold, marking the original image data as reference image data; Eliminating abnormal values ​​from all the reference image data in the video data; If the outliers are eliminated, the video data is divided into a plurality of segments; the reference image data in the same segment all match the sample image data in the same frame; In each of the segments, the original image data corresponding to the second probability with the highest value is screened out as the target image data.

7. The method according to claim 6, characterized in that The step of removing abnormal values ​​from all the reference image data in the video data comprises: Adding a slidable window to all the reference image data in the video data; If the reference image data located at non-midpoints of the window all match the sample image data of the same frame, and the sample image data matched by the reference image data located at the midpoint of the window is different from the sample image data matched by the reference image data located at the non-midpoints of the window, the reference image data located at the midpoint of the window will be eliminated as an outlier.

8. A device for capturing vehicle cleaning in a video based on depth features, characterized in that: include: Model determination module, used to determine the target detection network and the classifier built based on deep learning; A video data receiving module, used for receiving video data collected from a vehicle washing area; The video data contains multiple frames of original image data; A regional image data detection module, used for inputting the original image data into the target detection network to detect regional image data where the vehicle is located; A first probability calculation module, used in the classifier to extract target image features from the regional image data and calculate a first probability of cleaning the vehicle based on the target image features; A second probability calculation module is used to correct the first probability according to the target image feature in a representative dimension to obtain a second probability of cleaning the vehicle; The target image screening module is used to screen out part of the original image data according to the second probability as target image data representing the vehicle being cleaned.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for capturing vehicle cleaning based on depth features in a video as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for capturing vehicle cleaning based on depth features in a video according to any one of claims 1 to 7 is implemented.

Images