Vehicle verification method and device
The pre-trained neural network model processes video data and identifies the operation instructions executed by the vehicle, solving the problem of low efficiency in vehicle information verification in the prior art, and achieving fast and accurate vehicle information verification.
Patent Information
- Application Number
- CN201911077794.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-11-06
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-11-06
AI Technical Summary
The prior art is inefficient when verifying the authenticity of information about online car-hailing vehicles, and it is difficult to accurately judge the consistency of the operation instructions executed by the vehicle and the information input by the user.
The pre-trained neural network model is used to determine the vehicle's legality verification result through the object detection model and instruction recognition model, the video data sent by the user is processed, the operation instructions to be verified are identified, and the operation instructions are matched with the target operation instructions to determine the vehicle's legality verification result.
It improves the accuracy and efficiency of vehicle verification, can quickly and accurately judge the authenticity of vehicle information, and reduces the dependence of manual review.
Smart Images

Figure CN111860090B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of deep learning technology, and more specifically, to a vehicle verification method and device. Background Art
[0002] At present, with the rapid development of online ride-hailing, more and more users choose online ride-hailing as a means of travel, which makes users pay more and more attention to the safety of online ride-hailing. In order to improve the safety of online ride-hailing, when registering a vehicle, the platform needs to verify the real information of the vehicle, that is, to determine whether the vehicle information entered by the user is consistent with the real information, and perform corresponding business operations based on the verification results. Therefore, it is increasingly important to design a method to verify the vehicle. Summary of the invention
[0003] In view of this, the purpose of the present application is to provide a vehicle verification method and device to realize the authenticity verification of vehicle information.
[0004] In a first aspect, an embodiment of the present application provides a vehicle verification method, the method comprising:
[0005] In response to a vehicle verification request sent by a user terminal, a target operation instruction is selected from a pre-stored verification instruction set, and the target operation instruction is sent to the user terminal; the verification instruction set includes a plurality of operation instructions for instructing to perform different operations on the vehicle;
[0006] Receiving original video data sent by the user terminal based on the target operation instruction, inputting the original video data into a pre-trained neural network model for processing, and outputting the identified operation instruction to be verified;
[0007] Based on the matching result between the operation instruction to be verified and the target operation instruction, a vehicle legality verification result corresponding to the vehicle verification request is determined.
[0008] In an optional implementation, the verification instruction set includes a plurality of operation instructions for instructing different operations to be performed on the vehicle's lights.
[0009] In an optional implementation manner, the verification instruction set includes multiple types of the following operation instructions:
[0010] Turn on the left flash, turn on the right flash, turn on both flashes, turn on the headlights, turn on the headlights and turn on the left flash, turn on the headlights and turn on the right flash, turn on the headlights and turn on both flashes.
[0011] In an optional implementation, the pre-trained neural network model includes a target detection model and an instruction recognition model, and the inputting of the original video data into the pre-trained neural network model for processing and outputting the recognized operation instruction to be verified includes:
[0012] Input the original video data into the trained target detection model to obtain the target component area in each frame image corresponding to the original video data; the target component area is the area where the effective components of the different operations performed on the vehicle are located;
[0013] Based on the target component region in each acquired frame of image, generating intermediate video data containing the target component region;
[0014] The intermediate video data is input into the instruction recognition model to obtain the recognized operation instruction to be verified.
[0015] In an optional implementation, the target detection model includes a first convolutional neural network, a region recommendation network, a region mapping pooling layer, a fully connected network, a regression network and a first classifier;
[0016] The step of inputting the original video data into a trained target detection model to obtain a target component region in each frame image corresponding to the original video data includes:
[0017] For each frame of image corresponding to the original video data, the following process is performed:
[0018] Inputting the image into the first convolutional neural network for convolution processing to obtain first feature data of the image, wherein the first feature data is used to characterize local features of the image;
[0019] Inputting the first feature data into the region recommendation network and the region mapping pooling layer, processing the first feature data through the region recommendation network, and obtaining coordinate information of the candidate target region corresponding to the first feature data; inputting the coordinate information of the candidate target region into the region mapping pooling layer, mapping the coordinate information of the candidate target region onto the first feature data through the region mapping pooling layer, and obtaining second feature data corresponding to the candidate target region from the first feature data, wherein the second feature data is used to characterize local features of the candidate target region;
[0020] For each candidate region, the second feature data corresponding to the candidate target region is input into the fully connected network for processing, and the fully connected feature data corresponding to the second feature data is obtained, and the fully connected feature data is input into the regression network and the first classifier, and the fully connected feature data is processed by the regression network to generate regression data corresponding to the candidate target region, and the coordinate information of the candidate target region is changed based on the regression data; the fully connected feature data is processed by the first classifier to obtain the category and probability value of the candidate target region;
[0021] Based on the changed coordinate information of each candidate target region and the category and probability value corresponding to the candidate target region, the target component region corresponding to the image is obtained.
[0022] In an optional implementation, the first convolutional neural network includes at least one convolutional layer;
[0023] Inputting the image into the first convolutional neural network for convolution processing to obtain first feature data of the image includes:
[0024] The image is convolved by each convolution layer in the at least one convolution layer to obtain first feature data of the image data.
[0025] In an optional implementation, the region recommendation network includes a second convolutional neural network, a target region regression network, and a target region classification network; wherein n types of detection frames are provided in the region recommendation network, wherein the ratios of width to height of different detection frames are different, or the areas of different detection frames are different, and n is a positive integer;
[0026] The processing of the first feature data by a region recommendation network to obtain location information of a candidate target region corresponding to the first feature data of the image includes:
[0027] Inputting the first feature data into the second convolutional neural network for convolution processing to obtain first intermediate feature data corresponding to the first feature data, wherein the first intermediate feature data includes a plurality of first intermediate feature vectors, each of which corresponds to initial coordinate information of n types of detection boxes;
[0028] The first intermediate feature data are respectively input into the target area regression network and the target area classification network, and the first intermediate feature data are processed by the target area regression network to obtain second regression data of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data, and the target coordinate information of the detection frame is obtained based on the second regression data and the initial coordinate information of the detection frame; the first intermediate feature data are processed by the target area classification network to obtain second probability values and second categories of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data, and the position information of the candidate target area corresponding to the first feature data is obtained based on the second probability values of multiple detection frames, the second category and the target coordinate information of each detection frame.
[0029] In an optional implementation, the instruction recognition model includes: a third convolutional neural network, a first recurrent neural network, and a second classifier;
[0030] The step of inputting the intermediate video data into the instruction recognition model to obtain the recognized operation instruction to be verified includes:
[0031] Inputting the intermediate video data into the third convolutional neural network for convolution processing to obtain a third feature data sequence, wherein the third feature data sequence includes third feature data corresponding to each frame image in the intermediate video data, and an arrangement order of the third feature data in the third feature data sequence is the same as an arrangement order of each frame image in the intermediate video data;
[0032] Inputting the third feature data sequence into the first recurrent neural network, obtaining third intermediate feature data corresponding to each of the third feature data, wherein the third intermediate feature data is used to characterize the local features of any frame image in the intermediate video data;
[0033] Input each third intermediate feature data into the second classifier to obtain a detection result corresponding to each third intermediate feature data;
[0034] Based on the detection result corresponding to each third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data is obtained.
[0035] In an optional implementation, the step of inputting the third feature data sequence into the first recurrent neural network to obtain intermediate feature data corresponding to each of the third feature data includes:
[0036] According to the arrangement order of the third feature data, selecting a third feature data from the third feature data sequence as the current third feature data;
[0037] Inputting the current third feature data and the intermediate feature data corresponding to the previous third feature data into the first recurrent neural network to obtain the intermediate feature data corresponding to the current third feature data; wherein the intermediate feature data corresponding to the previous third feature data is obtained based on the previous third feature data and the first recurrent neural network;
[0038] Return to the step of selecting one third feature data from the third feature data sequence as the current third feature data according to the arrangement order of the third feature data, until the intermediate feature data of all the third feature data in the third feature data sequence is obtained.
[0039] In an optional implementation, the detection result includes a third probability value and a third category;
[0040] The obtaining, based on the detection result corresponding to each third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data comprises:
[0041] Setting a weight value for each third intermediate feature data in a preset manner;
[0042] Based on the third probability value, the third category and the set weight value corresponding to each of the third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data is determined.
[0043] In an optional implementation, after receiving the original video data sent by the user terminal based on the target operation instruction, before inputting the original video data into a pre-trained neural network model for processing and outputting the identified operation instruction to be verified, the method further includes:
[0044] Determine whether the duration of the original video data meets the set duration condition, and if so, input the original video data into a pre-trained neural network model;
[0045] If not satisfied, a return signal is sent to the user terminal, so that the user terminal re-sends the updated original video data based on the target operation instruction.
[0046] In a second aspect, an embodiment of the present application provides a vehicle verification device, the device comprising:
[0047] A response module, used to respond to the vehicle verification request sent by the user terminal, select a target operation instruction from a pre-stored verification instruction set, and send the target operation instruction to the user terminal; the verification instruction set includes a plurality of operation instructions for instructing to perform different operations on the vehicle;
[0048] A recognition module, used for receiving the original video data sent by the user terminal based on the target operation instruction, inputting the original video data into a pre-trained neural network model for processing, and outputting the recognized operation instruction to be verified;
[0049] A determination module is used to determine a vehicle legality verification result corresponding to the vehicle verification request based on a matching result between the operation instruction to be verified and the target operation instruction.
[0050] In an optional implementation, the verification instruction set includes a plurality of operation instructions for instructing different operations to be performed on the vehicle's lights.
[0051] In an optional implementation manner, the verification instruction set includes multiple types of the following operation instructions:
[0052] Turn on the left flash, turn on the right flash, turn on both flashes, turn on the headlights, turn on the headlights and turn on the left flash, turn on the headlights and turn on the right flash, turn on the headlights and turn on both flashes.
[0053] In an optional implementation, the pre-trained neural network model includes a target detection model and an instruction recognition model. When identifying the operation instruction to be verified corresponding to the original video data, the recognition module includes:
[0054] A target component region acquisition unit is used to input the original video data into a trained target detection model to acquire a target component region in each frame image corresponding to the original video data; the target component region is a region where the effective components of the different operations performed on the vehicle are located;
[0055] An intermediate video data generating unit, configured to generate intermediate video data containing the target component region based on the target component region in each acquired frame of image;
[0056] The unit for acquiring the operation instruction to be verified is used to input the intermediate video data into the instruction recognition model to acquire the recognized operation instruction to be verified.
[0057] In an optional implementation, the target detection model includes a first convolutional neural network, a region recommendation network, a region mapping pooling layer, a fully connected network, a regression network and a first classifier;
[0058] The target component region acquisition unit acquires the target component region in each frame image corresponding to the original video data by using the following steps:
[0059] For each frame of image corresponding to the original video data, the following process is performed:
[0060] Inputting the image into the first convolutional neural network for convolution processing to obtain first feature data of the image, wherein the first feature data is used to characterize local features of the image;
[0061] Inputting the first feature data into the region recommendation network and the region mapping pooling layer, processing the first feature data through the region recommendation network, and obtaining coordinate information of the candidate target region corresponding to the first feature data; inputting the coordinate information of the candidate target region into the region mapping pooling layer, mapping the coordinate information of the candidate target region onto the first feature data through the region mapping pooling layer, and obtaining second feature data corresponding to the candidate target region from the first feature data, wherein the second feature data is used to characterize local features of the candidate target region;
[0062] For each candidate region, the second feature data corresponding to the candidate target region is input into the fully connected network for processing, and the fully connected feature data corresponding to the second feature data is obtained, and the fully connected feature data is input into the regression network and the first classifier, and the fully connected feature data is processed by the regression network to generate regression data corresponding to the candidate target region, and the coordinate information of the candidate target region is changed based on the regression data; the fully connected feature data is processed by the first classifier to obtain the category and probability value of the candidate target region;
[0063] Based on the changed coordinate information of each candidate target region and the category and probability value corresponding to the candidate target region, the target component region corresponding to the image is obtained.
[0064] In an optional implementation, the first convolutional neural network includes at least one convolutional layer;
[0065] The target component region acquisition unit acquires the first feature data of the image by using the following steps:
[0066] The image is convolved by each convolution layer in the at least one convolution layer to obtain first feature data of the image data.
[0067] In an optional implementation, the region recommendation network includes a second convolutional neural network, a target region regression network, and a target region classification network; wherein n types of detection frames are provided in the region recommendation network, wherein the ratios of width to height of different detection frames are different, or the areas of different detection frames are different, and n is a positive integer;
[0068] The target component region acquisition unit acquires the position information of the candidate target region corresponding to the first feature data of the image by using the following steps:
[0069] Inputting the first feature data into the second convolutional neural network for convolution processing to obtain first intermediate feature data corresponding to the first feature data, wherein the first intermediate feature data includes a plurality of first intermediate feature vectors, each of which corresponds to initial coordinate information of n types of detection boxes;
[0070] The first intermediate feature data are respectively input into the target area regression network and the target area classification network, and the first intermediate feature data are processed by the target area regression network to obtain second regression data of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data, and the target coordinate information of the detection frame is obtained based on the second regression data and the initial coordinate information of the detection frame; the first intermediate feature data are processed by the target area classification network to obtain second probability values and second categories of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data, and the position information of the candidate target area corresponding to the first feature data is obtained based on the second probability values of multiple detection frames, the second category and the target coordinate information of each detection frame.
[0071] In an optional implementation, the instruction recognition model includes: a third convolutional neural network, a first recurrent neural network, and a second classifier;
[0072] The unit for acquiring the operation instruction to be verified acquires the identified operation instruction to be verified by using the following steps:
[0073] Inputting the intermediate video data into the third convolutional neural network for convolution processing to obtain a third feature data sequence, wherein the third feature data sequence includes third feature data corresponding to each frame image in the intermediate video data, and an arrangement order of the third feature data in the third feature data sequence is the same as an arrangement order of each frame image in the intermediate video data;
[0074] Inputting the third feature data sequence into the first recurrent neural network, obtaining third intermediate feature data corresponding to each of the third feature data, wherein the third intermediate feature data is used to characterize the local features of any frame image in the intermediate video data;
[0075] Input each third intermediate feature data into the second classifier to obtain a detection result corresponding to each third intermediate feature data;
[0076] Based on the detection result corresponding to each third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data is obtained.
[0077] In an optional implementation manner, the unit for acquiring the operation instruction to be verified acquires the intermediate feature data corresponding to each of the third feature data by using the following steps:
[0078] According to the arrangement order of the third feature data, selecting a third feature data from the third feature data sequence as the current third feature data;
[0079] Inputting the current third feature data and the intermediate feature data corresponding to the previous third feature data into the first recurrent neural network to obtain the intermediate feature data corresponding to the current third feature data; wherein the intermediate feature data corresponding to the previous third feature data is obtained based on the previous third feature data and the first recurrent neural network;
[0080] Return to the step of selecting one third feature data from the third feature data sequence as the current third feature data according to the arrangement order of the third feature data, until the intermediate feature data of all the third feature data in the third feature data sequence is obtained.
[0081] In an optional implementation, the detection result includes a third probability value and a third category;
[0082] The to-be-verified operation instruction acquisition unit obtains the to-be-verified operation instruction corresponding to the intermediate video data by using the following steps:
[0083] Setting a weight value for each third intermediate feature data in a preset manner;
[0084] Based on the third probability value, the third category and the set weight value corresponding to each of the third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data is determined.
[0085] In an optional implementation, after receiving the original video data sent by the user terminal based on the target operation instruction, before inputting the original video data into a pre-trained neural network model for processing and outputting the identified operation instruction to be verified, the device further includes:
[0086] A judgment module, used to judge whether the duration of the original video data meets the set duration condition, and if so, input the original video data into a pre-trained neural network model;
[0087] If not satisfied, a return signal is sent to the user terminal, so that the user terminal re-sends the updated original video data based on the target operation instruction.
[0088] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the vehicle verification method as described in any one of the first aspects.
[0089] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the vehicle verification method as described in any one of the first aspects are executed.
[0090] The embodiment of the present application responds to a vehicle verification request sent by a user terminal, selects a target operation instruction from a pre-stored verification instruction set, sends the target operation instruction to the user terminal, receives original video data sent by the user terminal based on the target operation instruction, inputs the original video data into a pre-trained neural network model for processing, outputs the identified operation instruction to be verified, determines the vehicle legitimacy verification result corresponding to the vehicle verification request through the matching result of the operation instruction to be verified and the target operation instruction, thereby realizing the verification of the authenticity of the vehicle information. BRIEF DESCRIPTION OF THE DRAWINGS
[0091] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0092] Figure 1 A schematic diagram of the architecture of a service system provided by an embodiment of the present application is shown;
[0093] Figure 2 A schematic diagram showing exemplary hardware and software components of an electronic device 200 of a server 110, a service requester terminal 130, and a service provider terminal 140 that can implement the concept of the present application according to some embodiments of the present application;
[0094] Figure 3 A flow chart of a vehicle verification method provided by an embodiment of the present application is shown;
[0095] Figure 4 A flowchart showing a specific method for determining an operation instruction to be verified corresponding to original video data based on a neural network model in a vehicle verification method provided in an embodiment of the present application;
[0096] Figure 5 A flowchart showing a specific method for obtaining a target component area in each frame image corresponding to original video data in a vehicle verification method provided in an embodiment of the present application;
[0097] Figure 6 A flowchart showing a specific method for determining location information of a candidate target area in a vehicle verification method provided in an embodiment of the present application;
[0098] Figure 7 A flowchart showing a specific method for obtaining an operation instruction to be verified in a vehicle verification method provided in an embodiment of the present application;
[0099] Figure 8 A schematic diagram of the structure of a command recognition model in a vehicle verification method provided in an embodiment of the present application is shown;
[0100] Fig. 9 A schematic structural diagram of a vehicle verification device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0101] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.
[0102] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0103] In order to enable those skilled in the art to use the contents of this application, the following implementation is given in conjunction with the specific application scenario "verifying the legitimacy of a vehicle". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is mainly described around verifying the legitimacy of a vehicle, it should be understood that this is only an exemplary embodiment. This application can be applied to verify any vehicle. For example, vehicles may include taxis, private cars, rideshares, buses, trains, bullet trains, high-speed railways, subways, ships, airplanes, or unmanned vehicles, or any combination thereof.
[0104] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0105] One aspect of the present application relates to a vehicle verification method. The method can select a target operation instruction from a pre-stored verification instruction set in response to a vehicle verification request sent by a user terminal, send the target operation instruction to the user terminal, receive raw video data sent by the user terminal based on the target operation instruction, input the raw video data into a pre-trained neural network model for processing, output the identified operation instruction to be verified, and through the matching result of the operation instruction to be verified and the target operation instruction, the vehicle legality verification result corresponding to the vehicle verification request can be determined more accurately, thereby improving the accuracy of vehicle verification.
[0106] It is worth noting that before the present application was filed, when verifying the authenticity of vehicle information, it was necessary to manually review the authenticity of the vehicle information. However, the method of manually reviewing the authenticity of vehicle information has the problem of low review efficiency. The vehicle verification method proposed in the present application determines the vehicle legality verification result corresponding to the vehicle verification request based on the matching result of the operation instruction to be verified and the target operation instruction in the received original video data, thereby improving the efficiency of vehicle verification.
[0107] Figure 1 It is a block diagram of a system 100 in a scenario in which the vehicle verification method of some embodiments of the present application is applied. For example, the system 100 may be an online verification platform for verifying vehicles such as taxis, express trains, carpools, buses, etc. The system 100 may include one or more of a server 110, a network 120, a service requester terminal 130, a service provider terminal 140, and a database 150, and the server 110 may include a processor for executing instruction operations. The vehicle verification method provided in the embodiment of the present application can be applied to the server 110 in the above-mentioned system 100, and the relevant operation instructions can be executed by the processor.
[0108] Figure 2 A schematic diagram showing exemplary hardware and software components of an electronic device 200 that can implement the server 110 of the present invention according to some embodiments of the present invention. For example, a processor can be used on the electronic device 200 and used to execute the functions in the present invention.
[0109] The electronic device 200 can be a general-purpose computer or a special-purpose computer, both of which can be used to implement the vehicle verification method of the present application. Although only one computer is shown in the present application, for convenience, the functions described in the present application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.
[0110] For example, the electronic device 200 may include a network port 210 connected to a network, one or more processors 220 for executing program instructions, a communication bus 230, and storage media 240 in different forms, such as disks, ROMs, or RAMs, or any combination thereof. Exemplarily, the computer platform may also include program instructions stored in ROMs, RAMs, or other types of non-transitory storage media, or any combination thereof. The method of the present application may be implemented according to these program instructions. The electronic device 200 also includes an input / output (I / O) interface 250 between the computer and other input / output devices (e.g., keyboards, display screens).
[0111] For ease of description, only one processor is described in the electronic device 200. However, it should be noted that the electronic device 200 in the present application may also include multiple processors, so the steps performed by one processor described in the present application may also be performed jointly or individually by multiple processors. For example, if the processor of the electronic device 200 performs step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the processor performs step A and the second processor performs step B, or the processor and the second processor perform steps A and B together.
[0112] Embodiment 1
[0113] Figure 3 A flow chart of a vehicle verification method provided in an embodiment of the present application is shown, including S301-S303.
[0114] S301: In response to a vehicle verification request sent by a user terminal, a target operation instruction is selected from a pre-stored verification instruction set, and the target operation instruction is sent to the user terminal; the verification instruction set includes multiple operation instructions for instructing to perform different operations on the vehicle.
[0115] In a specific implementation, the vehicle verification method is applied to a server, and the user terminal can send a vehicle verification request to the server. After receiving the vehicle verification request sent by the user terminal, the server selects an operation instruction from a pre-stored verification instruction set as a target operation instruction, and sends the target operation instruction to the user terminal. In the embodiment of the present application, the verification instruction set includes a plurality of operation instructions for instructing the vehicle to perform different operations. For example, instructing the vehicle to perform different operations can be instructing the vehicle's lights to perform different operations, or / and instructing the vehicle's horn to perform different operations, or / and instructing the vehicle's wipers to perform different operations, etc.
[0116] Exemplarily, the verification instruction set includes a plurality of operation instructions for performing different operations on the vehicle's lights.
[0117] For example, if instructing the vehicle to perform different operations is to instruct the vehicle's lights to perform different operations, the verification instruction set includes multiple of the following operation instructions: turn on the left flash, turn on the right flash, turn on the double flash, turn on the headlights, turn on the headlights and turn on the left flash, turn on the headlights and turn on the right flash, turn on the headlights and turn on the double flash. The verification operation instruction set may also include turning on the left flash first and then the right flash, turning on the right flash first and then the double flash, etc. In specific implementation, the operation instructions in the verification instruction set may be set according to actual conditions, and the embodiments of the present application do not specifically limit this.
[0118] S302: Receive original video data sent by the user terminal based on the target operation instruction, input the original video data into a pre-trained neural network model for processing, and output the identified operation instruction to be verified.
[0119] In specific implementation, the user terminal captures original video data based on the received target operation instructions and sends the original video data to the server. The server inputs the original video data into a pre-trained neural network model for processing and outputs the operation instructions to be verified corresponding to the original video data.
[0120] Exemplarily, the pre-trained neural network model includes a target detection model and a command recognition model. For details, see Figure 4 As shown, an embodiment of the present application provides a flowchart of a specific method for determining the operation instructions to be verified corresponding to the original video data based on a neural network model.
[0121] Exemplarily, the original video data is input into a pre-trained neural network model for processing, and the identified operation instructions to be verified are output, including:
[0122] S401, inputting the original video data into the trained target detection model to obtain the target component area in each frame image corresponding to the original video data; the target component area is the area where the effective components of different operations performed on the vehicle are located.
[0123] Exemplarily, the target component area may be an area where the headlights are located, an area where the wipers are located, etc. Specifically, the target component area is an area where the functional components of the vehicle that perform different operations are located. For example, if the verification instruction set includes an operation instruction that instructs the headlights of the vehicle to perform different operations, the target component area is an area where the headlights are located. Specifically, continuing with the example in S301, the target component area in the embodiment of the present application is an area where the headlights are located.
[0124] Exemplarily, the target detection model may be a Faster R-CNN model, and the target detection area of each frame image corresponding to the original video data is obtained by the Faster R-CNN model. Among them, the target detection model may be a model for obtaining a preset target area in the image, and the embodiments of the present application are only exemplary.
[0125] S402: Generate intermediate video data containing the target component region based on the target component region in each acquired frame of image.
[0126] For example, an image of the target component region can be selected from each frame of the original video data, the selected image carries a timestamp, and the selected images of the target component region are arranged in ascending order of the timestamp to generate intermediate video data containing the target component region. Alternatively, the selected image carries time information, and the selected images of the target component region are arranged according to the time information to generate intermediate video data containing the target component region.
[0127] S403, inputting the intermediate video data into the instruction recognition model to obtain the recognized operation instruction to be verified.
[0128] Exemplarily, the intermediate video data is video data including the target component area, and the intermediate video data is video data obtained after performing area selection in the original video data. In the embodiment of the present application, based on the intermediate video data and the instruction recognition model, the operation instruction to be verified is determined, which can improve the efficiency of the instruction recognition model and improve the accuracy of recognition.
[0129] As an optional embodiment, the target detection model includes a first convolutional neural network, a region recommendation network, a region mapping pooling layer, a fully connected network, a regression network and a first classifier.
[0130] See also Figure 5 The flowchart of a specific method for obtaining the target component area in each frame image corresponding to the original video data is shown. Specifically, the original video data is input into the trained target detection model to obtain the target component area in each frame image corresponding to the original video data, including S501-S505. The specific process is as follows:
[0131] S501, for each frame of image corresponding to the original video data, perform the following process:
[0132] In the embodiment of the present application, each frame of the original video data is processed to obtain the target component area of each frame of the original video data. In particular, a frame of the original video data can be obtained at intervals of a first preset time, and then multiple images can be obtained from the original video data. In particular, the first preset time can be 0.1 seconds, 0.2 seconds, etc., and the specific first preset time can be set according to actual conditions.
[0133] S502: Input the image into a first convolutional neural network for convolution processing to obtain first feature data of the image, wherein the first feature data is used to characterize local features of the image.
[0134] In the embodiment of the present application, the first convolutional neural network performs convolution processing on the image to obtain first feature data of the image, that is, extracts local features of the image.
[0135] Exemplarily, the first convolutional neural network includes at least one convolutional layer; the image is input into the first convolutional neural network for convolution processing to obtain first feature data of the image, including: convolution processing is performed on the image through each convolutional layer in at least one convolutional layer to obtain first feature data of the image data.
[0136] For example, the first convolutional neural network can be a lightweight mobilenet_v2 network, a VGG16 network, etc. In the embodiment of the present application, the mobilenet_v2 network can be selected to perform convolution processing on the image to obtain the first feature data of the image.
[0137] S503, input the first feature data into the region recommendation network and the region mapping pooling layer, process the first feature data through the region recommendation network, and obtain the coordinate information of the candidate target region corresponding to the first feature data; input the coordinate information of the candidate target region into the region mapping pooling layer, map the coordinate information of the candidate target region onto the first feature data through the region mapping pooling layer, and obtain the second feature data corresponding to the candidate target region from the first feature data, wherein the second feature data is used to characterize the local features of the candidate target region.
[0138] Exemplarily, the region proposal network may be a region proposal network (RPN), and the region mapping pooling layer may be RoI pooling. Exemplarily, the first feature data is input into the RPN and RoIpooling, and the RPN processes the first feature data to obtain the coordinate information of the candidate target region corresponding to the first feature data; the coordinate information of the candidate target region is input into the RoI pooling, and the RoI pooling maps the coordinate information of the candidate target region onto the first feature data, and obtains the second feature data corresponding to the candidate target region from the first feature data, that is, obtains the local features corresponding to the target candidate region.
[0139] Exemplarily, the region recommendation network may include a second convolutional neural network, a target region regression network, and a target region classification network; wherein n types of detection frames are provided in the region recommendation network, wherein the ratios of width to height of different detection frames are different, or the areas of different detection frames are different, and n is a positive integer.
[0140] Exemplarily, in the embodiment of the present application, the ratio of the width to the height of the detection frame is set to a ratio greater than 1. For example, the value of n can be 9, the ratio of the width to the height includes 2:1, 3:1, 4:1; the area of the detection frame includes 128, 192, 256; then the 9 types of detection frames include a detection frame with a ratio of 2:1 and an area of 128, a detection frame with a ratio of 3:1 and an area of 128, a detection frame with a ratio of 4:1 and an area of 128; a detection frame with a ratio of 2:1 and an area of 192, a detection frame with a ratio of 3:1 and an area of 192, a detection frame with a ratio of 4:1 and an area of 192; a detection frame with a ratio of 2:1 and an area of 256, a detection frame with a ratio of 3:1 and an area of 256, and a detection frame with a ratio of 4:1 and an area of 256. Among them, the number of detection frames, the ratio of the width to the height corresponding to the detection frame, and the area of the detection frame can be set according to actual needs, and the embodiment of the present application does not specifically limit this.
[0141] See also Figure 6 A flowchart of a specific method for determining location information of a candidate target area in a vehicle verification method is shown. Specifically, the first feature data is processed by a region recommendation network to obtain the location information of the candidate target area corresponding to the first feature data of the image, including S601-S603.
[0142] The specific process is as follows:
[0143] S601, inputting the first feature data into the second convolutional neural network for convolution processing to obtain first intermediate feature data corresponding to the first feature data, wherein the first intermediate feature data includes a plurality of first intermediate feature vectors, each of which corresponds to initial coordinate information of n types of detection boxes;
[0144] S602, input the first intermediate feature data into the target area regression network and the target area classification network respectively, process the first intermediate feature data through the target area regression network, obtain the second regression data of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data, and obtain the target coordinate information of the detection frame based on the second regression data and the initial coordinate information of the detection frame; process the first intermediate feature data through the target area classification network, obtain the second probability value and the second category of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data, and obtain the position information of the candidate target area corresponding to the first feature data by using the second probability values, the second category and the target coordinate information of each detection frame of the multiple detection frames corresponding to the first intermediate feature data.
[0145] For S601: Exemplarily, the second convolutional neural network may include at least one second convolutional layer, and each convolutional layer in the at least one second convolutional layer performs convolution processing on the first feature data to obtain first intermediate feature data corresponding to the first feature data. The first intermediate feature data includes multiple first intermediate feature vectors, and each first intermediate feature vector corresponds to initial coordinate information of n types of detection frames, that is, if the first intermediate feature data includes m first intermediate feature vectors, the number of detection frames corresponding to the first intermediate feature data is m times n, and m is a positive integer.
[0146] For S602: Exemplarily, the target area regression network may include at least one third convolutional layer, and each convolutional layer in the at least one third convolutional layer performs convolution processing on the first intermediate feature data respectively to obtain second regression data of each detection frame in the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data, and obtains target coordinate information of the detection frame based on the second regression data of each detection frame in the first intermediate feature data, that is, obtains target coordinate information of m times n detection frames.
[0147] For S602: Exemplarily, the target area classification network includes at least one fourth convolution layer, a first reshape layer, a third classifier, and a second reshape layer. The first intermediate feature data is processed respectively by at least one fourth convolution layer, a first Reshape layer, a third classifier, and a second Reshape layer in the target area classification network to obtain a second probability value and a second category of each detection box in the n types of detection boxes corresponding to each first intermediate feature vector in the first intermediate feature data.
[0148] For S602: Exemplarily, after obtaining the second probability values, second categories and target coordinate information of each detection frame of multiple detection frames, the position information of the candidate target area corresponding to the first feature data can be obtained based on the second probability values, second categories and target coordinate information of each detection frame of multiple detection frames. Exemplarily, the process of obtaining the position information of the candidate target area corresponding to the first feature data based on the second probability values, second categories and target coordinate information of each detection frame of multiple detection frames can be: if the second probability value is the probability that the detection frame corresponds to the foreground category, k detection frames with larger second probability values are selected from m multiplied by n detection frames, and the k detection frames are screened by the non-maximum suppression (nms) mechanism to obtain h detection frames, and the target coordinate information corresponding to the h detection frames is the position information of the candidate target area corresponding to the first feature data, wherein k and h are positive integers, and the values of k and h can be set according to actual conditions. In specific implementation, if there is no preset target component area in the original video data, the number of candidate target areas corresponding to the first feature data is zero, and the number of screened detection frames is zero, that is, h is zero.
[0149] S504, for each candidate area, input the second feature data corresponding to the candidate target area into the fully connected network for processing, obtain the fully connected feature data corresponding to the second feature data, input the fully connected feature data into the regression network and the first classifier, process the fully connected feature data through the regression network, generate the regression data corresponding to the candidate target area, and change the coordinate information of the candidate target area based on the regression data; process the fully connected feature data through the first classifier to obtain the category and probability value of the candidate target area.
[0150] Exemplarily, the regression network processes the fully connected data to obtain the regression data corresponding to the candidate region, and the regression data is the offset of the coordinate information of the target region. The coordinate information of the candidate target region is changed by the obtained regression data to obtain the changed coordinate information of the candidate target region, that is, the target coordinate information of the candidate target region is obtained.
[0151] Exemplarily, the first classifier processes the fully connected feature data to obtain the category and probability value of the candidate target area, the category includes foreground and background, and the probability value is the credibility of the candidate target area being the foreground or the background. For example, if the category corresponding to the candidate area A is foreground and the probability is 70%, then the credibility of the candidate area being the foreground is 70%, that is, the credibility of the candidate area being the background is 30%.
[0152] S505 , obtaining a target component region corresponding to the image based on the changed coordinate information of each candidate target region and the category and probability value corresponding to the candidate target region.
[0153] Exemplarily, if the number of candidate target areas is h, the h candidate target areas are screened through the nms mechanism to obtain g target areas to be tested, and the target area to be tested with the largest probability value is selected from the g target areas to be tested as the target component area of the image, where g is a positive integer less than or equal to h.
[0154] As an optional embodiment, the instruction recognition model may include: a third convolutional neural network, a first recurrent neural network, and a second classifier.
[0155] See also Figure 7 The flowchart of a specific method for obtaining an operation instruction to be verified in a vehicle verification method shown in FIG. 1 is a flowchart of a specific method for obtaining an operation instruction to be verified in a vehicle verification method. Specifically, the intermediate video data is input into the instruction recognition model to obtain the recognized operation instruction to be verified, including S701-S704. The specific situation is as follows:
[0156] S701, inputting the intermediate video data into a third convolutional neural network for convolution processing to obtain a third feature data sequence, wherein the third feature data sequence includes third feature data corresponding to each frame of the intermediate video data, and an arrangement order of the third feature data in the third feature data sequence is the same as an arrangement order of each frame of the intermediate video data;
[0157] S702, inputting the third feature data sequence into the first recurrent neural network, and obtaining third intermediate feature data corresponding to each third feature data, wherein the third intermediate feature data is used to characterize the local features of any frame image in the intermediate video data;
[0158] S703, input each third intermediate feature data into the second classifier to obtain a detection result corresponding to each third intermediate feature data;
[0159] S704: Based on the detection result corresponding to each third intermediate feature data, obtain the operation instruction to be verified corresponding to the intermediate video data.
[0160] For example, see Figure 8A schematic diagram of the structure of a command recognition model in a vehicle verification method is shown. Figure 8 In the example, the instruction recognition model includes a third convolutional neural network 81, a first recurrent neural network 82, and a second classifier 83. The intermediate video data is input into the third convolutional neural network for convolution processing to obtain a third feature data sequence, which includes x third feature data, where x is a positive integer, that is, the third feature data sequence includes third feature data s 1 , the third characteristic data s 2 , ..., the third characteristic data s x , input the third feature data sequence into the first recurrent neural network, obtain the third intermediate feature data corresponding to each third feature data, and obtain s 1 The corresponding third intermediate feature data y 1 、s 2 The corresponding third intermediate feature data y 2 ,…,s x The corresponding third intermediate feature data y x , the third intermediate feature data y 1 , the third intermediate feature data y 2 , ..., the third intermediate feature data y x Input them into the second classifier respectively to obtain the detection result corresponding to each third intermediate feature data, that is, y 1 The corresponding test results, y 2 The corresponding test results, ..., y x The corresponding test results are 1 The corresponding test results, y 2 The corresponding test results, ..., y x The corresponding detection results are weighted and summed to obtain the probability and category corresponding to the intermediate video data, and then the operation instruction to be verified is determined according to the probability and category corresponding to the intermediate video data.
[0161] For S701, illustratively, the intermediate video data is input into the third convolutional neural network for convolution processing, and then, at every second preset time interval, a frame of image is obtained from the intermediate video data, and the obtained image is input into the third convolutional neural network to obtain the third feature data of the image, and the obtained multiple third feature data constitute a third feature data sequence. Among them, the second preset time can be the same as the first preset time, or it can be different, and the second preset time can be set according to actual conditions.
[0162] For S702, illustratively, the first recurrent neural network may be a long short-term memory (LSTM) network. The third feature data sequence is input into the LSTM network to obtain third intermediate feature data corresponding to each third feature data.
[0163] For S703, illustratively, the detection result includes a third probability value and a third category. Each third intermediate feature data is input into the second classifier to obtain a third probability value and a third category corresponding to each third intermediate feature data.
[0164] For S704, illustratively, based on the detection result corresponding to each third intermediate feature data, obtaining the to-be-verified operation instruction corresponding to the intermediate video data includes:
[0165] Setting a weight value for each third intermediate feature data in a preset manner;
[0166] Based on the third probability value, the third category and the set weight value corresponding to each third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data is determined.
[0167] Exemplarily, first, the third probability value corresponding to each third intermediate feature data is multiplied by the weight value corresponding to the third intermediate feature data, and then, based on the same category, the values obtained after the product of each third intermediate feature data are added, that is, the third category corresponding to the intermediate video data and the probability value of the intermediate video data corresponding to the third category are obtained, and the category corresponding to the maximum probability value of the intermediate video data is determined as the operation instruction to be verified corresponding to the intermediate video data. The third category corresponds to a plurality of operation instructions included in the verification instruction set. For example, if the verification instruction set includes turning on the left flash, turning on the right flash, turning on the double flash, and turning on the headlights, then the third category includes: turning on the left flash, turning on the right flash, turning on the double flash, and turning on the headlights, and the third probability value corresponding to the third intermediate feature data includes the probability of turning on the left flash, the probability of turning on the right flash, the probability of turning on the double flash, and the probability of turning on the headlights. Taking the category of turning on the left flash as an example, the probability of turning on the left flash corresponding to each third intermediate feature data is multiplied by the weight value of the third intermediate feature data, and the values obtained after the product of each third intermediate feature data are added to obtain the probability value of turning on the left flash corresponding to the intermediate video data, and then the probability values of turning on the right flash, turning on the double flash, and turning on the headlights corresponding to the intermediate video data can be obtained. The category corresponding to the maximum probability value among the probability values of turning on the left flash, the probability values of turning on the right flash, the probability values of turning on the double flash, and the probability values of turning on the headlights is determined as the operation instruction to be verified corresponding to the intermediate video data. For example, if the probability value of turning on the headlights is the largest, then the operation instruction to be verified corresponding to the intermediate video data is turning on the headlights.
[0168] Exemplarily, if each frame of the intermediate video data carries a timestamp, the third feature data corresponding to the image also carries a timestamp. Since in the third feature data sequence, the larger the timestamp of the third feature data, the higher the importance of the third feature data, a larger weight can be set for the third intermediate feature data corresponding to the third feature data with high importance.
[0169] Exemplarily, the preset method may be to divide the third intermediate feature data into different levels according to the timestamp, and to set different weights based on the different levels. Alternatively, the preset method may be to sort the third intermediate feature data from small to large according to the value of the timestamp, and to set a weight value for each third intermediate feature data in an ascending manner according to the arrangement order of the third intermediate feature data by the weight value. For example, if the number of images corresponding to the intermediate video data is 30, and the 30 images are sorted from small to large according to the value of the timestamp, then the third intermediate feature data sequence corresponding to the sorted 30 images is divided into 5 levels, and 10 third intermediate feature data from the third intermediate feature data corresponding to the 1st image to the third intermediate feature data corresponding to the 10th image are divided into the first level, and 5 third intermediate feature data from the third intermediate feature data corresponding to the 11th image to the third intermediate feature data corresponding to the 15th image are divided into the second level, and 5 third intermediate feature data from the third intermediate feature data corresponding to the 16th image to the third intermediate feature data corresponding to the 20th image are divided into the third level, and the 21st image is divided into the third level. The third intermediate feature data corresponding to the image to the third intermediate feature data corresponding to the 25th image are divided into the 4th level, and the third intermediate feature data corresponding to the 26th image to the third intermediate feature data corresponding to the 30th image are divided into the 5th level, and the weight value corresponding to the third intermediate feature data in the 1st level is set to 0, the weight value corresponding to the third intermediate feature data in the 2nd level is set to 0.01, the weight value corresponding to the third intermediate feature data in the 3rd level is set to 0.04, the weight value corresponding to the third intermediate feature data in the 4th level is set to 0.05, and the weight value corresponding to the third intermediate feature data in the 5th level is set to 0.1. Alternatively, the weight values are set for the third intermediate feature data corresponding to the 30 images in a manner of increasing the weight values in sequence.
[0170] As an optional embodiment, the third feature data sequence is input into the first recurrent neural network to obtain the intermediate feature data corresponding to each third feature data, including:
[0171] According to the arrangement order of the third feature data, select one third feature data from the third feature data sequence as the current third feature data;
[0172] Inputting the current third feature data and the intermediate feature data corresponding to the previous third feature data into the first recurrent neural network to obtain the intermediate feature data corresponding to the current third feature data; wherein the intermediate feature data corresponding to the previous third feature data is obtained based on the previous third feature data and the first recurrent neural network;
[0173] Return to the step of selecting one third feature data from the third feature data sequence as the current third feature data according to the arrangement order of the third feature data, until the intermediate feature data of all the third feature data in the third feature data sequence are obtained.
[0174] Exemplarily, if the third feature data sequence includes 5 third feature data, namely, third feature data one, third feature data two, third feature data three, third feature data four, and third feature data five, the third feature data one is input into the first recurrent neural network to obtain the intermediate feature data one corresponding to the third feature data one; the third feature data two and the intermediate feature data one are input into the first recurrent neural network to obtain the intermediate feature data two corresponding to the third feature data two, until the intermediate feature data corresponding to each third feature data in the third feature data sequence is obtained, that is, the intermediate feature data three, the intermediate feature data four, and the intermediate feature data five are obtained.
[0175] S303: Based on the matching result between the operation instruction to be verified and the target operation instruction, determine the vehicle legality verification result corresponding to the vehicle verification request.
[0176] Exemplarily, if the matching result of the operation instruction to be verified and the target operation instruction are the same, then the vehicle legitimacy verification result corresponding to the vehicle verification request is determined to be a successful verification; if the matching result of the operation instruction to be verified and the target operation instruction are different, then the vehicle legitimacy verification result corresponding to the vehicle verification request is determined to be a failed verification.
[0177] As an optional embodiment, after receiving the original video data sent by the user terminal based on the target operation instruction, before inputting the original video data into the pre-trained neural network model for processing and outputting the identified operation instruction to be verified, the method further includes:
[0178] Determine whether the duration of the original video data meets the set duration condition. If so, input the original video data into the pre-trained neural network model; if not, send a return signal to the user end, so that the user end re-sends the updated original video data based on the target operation instruction.
[0179] Exemplarily, determining whether the duration of the original video data satisfies the set duration condition includes: determining whether the duration of the original video data is equal to the set duration, if so, satisfying the set duration condition. Alternatively, determining whether the duration of the original video data is greater than or equal to the set duration, if so, satisfying the set duration condition, and filtering the images of the original video data, selecting images from large to small timestamps, until the duration of the original video data composed of the selected images is equal to the set duration.
[0180] The embodiment of the present application responds to a vehicle verification request sent by a user terminal, selects a target operation instruction from a pre-stored verification instruction set, sends the target operation instruction to the user terminal, and receives original video data sent by the user terminal based on the target operation instruction, inputs the original video data into a pre-trained neural network model for processing, outputs an identified operation instruction to be verified, and determines a vehicle legality verification result corresponding to the vehicle verification request through a matching result between the operation instruction to be verified and the target operation instruction, thereby realizing authenticity verification of the vehicle information. Moreover, by verifying the authenticity of the vehicle through a neural network model, the efficiency of vehicle verification can be improved.
[0181] Based on the same inventive concept, a vehicle verification device corresponding to the vehicle verification method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned vehicle verification method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0182] Embodiment 2
[0183] Reference Fig. 9 As shown, it is a structural diagram of a vehicle verification device provided in the second embodiment of the present application, the device includes: a response module 91, an identification module 92, and a determination module 93; wherein,
[0184] The response module 91 is used to respond to the vehicle verification request sent by the user terminal, select a target operation instruction from a pre-stored verification instruction set, and send the target operation instruction to the user terminal; the verification instruction set includes a plurality of operation instructions for instructing to perform different operations on the vehicle;
[0185] The recognition module 92 is used to receive the original video data sent by the user terminal based on the target operation instruction, input the original video data into the pre-trained neural network model for processing, and output the recognized operation instruction to be verified;
[0186] The determination module 93 is used to determine the vehicle legality verification result corresponding to the vehicle verification request based on the matching result between the operation instruction to be verified and the target operation instruction.
[0187] In an optional implementation, the verification instruction set includes a plurality of operation instructions for instructing different operations to be performed on the vehicle's lights.
[0188] In an optional implementation, the verification instruction set includes multiple of the following operation instructions:
[0189] Turn on the left flash, turn on the right flash, turn on both flashes, turn on the headlights, turn on the headlights and turn on the left flash, turn on the headlights and turn on the right flash, turn on the headlights and turn on both flashes.
[0190] In an optional implementation, the pre-trained neural network model includes a target detection model and an instruction recognition model. When identifying the operation instruction to be verified corresponding to the original video data, the recognition module includes:
[0191] A target component region acquisition unit is used to input the original video data into the trained target detection model to acquire the target component region in each frame image corresponding to the original video data; the target component region is the region where the effective components of different operations performed on the vehicle are located;
[0192] An intermediate video data generating unit, used for generating intermediate video data containing a target component region based on the target component region in each frame of the acquired image;
[0193] The unit for acquiring the operation instruction to be verified is used to input the intermediate video data into the instruction recognition model to acquire the recognized operation instruction to be verified.
[0194] In an optional implementation, the target detection model includes a first convolutional neural network, a region recommendation network, a region mapping pooling layer, a fully connected network, a regression network, and a first classifier;
[0195] The target component region acquisition unit acquires the target component region in each frame image corresponding to the original video data by using the following steps:
[0196] For each frame of the original video data, the following process is performed:
[0197] Inputting the image into a first convolutional neural network for convolution processing to obtain first feature data of the image, wherein the first feature data is used to characterize local features of the image;
[0198] Input the first feature data into the region recommendation network and the region mapping pooling layer, process the first feature data through the region recommendation network, and obtain the coordinate information of the candidate target region corresponding to the first feature data; input the coordinate information of the candidate target region into the region mapping pooling layer, map the coordinate information of the candidate target region onto the first feature data through the region mapping pooling layer, and obtain the second feature data corresponding to the candidate target region from the first feature data, wherein the second feature data is used to characterize the local features of the candidate target region;
[0199] For each candidate region, the second feature data corresponding to the candidate target region is input into the fully connected network for processing, and the fully connected feature data corresponding to the second feature data is obtained. The fully connected feature data is input into the regression network and the first classifier, and the fully connected feature data is processed by the regression network to generate regression data corresponding to the candidate target region, and the coordinate information of the candidate target region is changed based on the regression data; the fully connected feature data is processed by the first classifier to obtain the category and probability value of the candidate target region;
[0200] Based on the changed coordinate information of each candidate target region and the category and probability value corresponding to the candidate target region, the target component region corresponding to the image is obtained.
[0201] In an optional implementation, the first convolutional neural network includes at least one convolutional layer;
[0202] The target component region acquisition unit acquires the first feature data of the image by using the following steps:
[0203] The image is convolved by each convolution layer in at least one convolution layer to obtain first feature data of the image data.
[0204] In an optional implementation, the region recommendation network includes a second convolutional neural network, a target region regression network, a target region classification network, and a target region determination network; wherein n types of detection frames are provided in the region recommendation network, wherein the ratios of width to height of different detection frames are different, or the areas of different detection frames are different, and n is a positive integer;
[0205] The target component region acquisition unit acquires the position information of the candidate target region corresponding to the first feature data of the image by using the following steps:
[0206] Inputting the first feature data into a second convolutional neural network for convolution processing to obtain first intermediate feature data corresponding to the first feature data, wherein the first intermediate feature data includes a plurality of first intermediate feature vectors, and each first intermediate feature vector corresponds to initial coordinate information of n types of detection frames;
[0207] The first intermediate feature data are respectively input into the target area regression network and the target area classification network, the first intermediate feature data are processed by the target area regression network, the second regression data of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data is obtained, and the second regression data and the initial coordinate information of the detection frame are input into the target area determination network for processing to obtain the target coordinate information of the detection frame; the first intermediate feature data are processed by the target area classification network, the second probability value and the second category of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data are obtained, the second probability values and the second categories of the multiple detection frames corresponding to the first intermediate feature data are input into the target area determination network for processing to obtain the position information of the candidate target area corresponding to the first feature data.
[0208] In an optional implementation, the instruction recognition model includes: a third convolutional neural network, a first recurrent neural network, and a second classifier;
[0209] The unit for obtaining the operation instruction to be verified obtains the identified operation instruction to be verified by using the following steps:
[0210] Inputting the intermediate video data into a third convolutional neural network for convolution processing to obtain a third feature data sequence, wherein the third feature data sequence includes third feature data corresponding to each frame image in the intermediate video data, and an arrangement order of the third feature data in the third feature data sequence is the same as an arrangement order of each frame image in the intermediate video data;
[0211] Inputting the third feature data sequence into the first recurrent neural network, obtaining third intermediate feature data corresponding to each third feature data, wherein the third intermediate feature data is used to characterize the local features of any frame image in the intermediate video data;
[0212] Input each third intermediate feature data into the second classifier to obtain a detection result corresponding to each third intermediate feature data;
[0213] Based on the detection result corresponding to each third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data is obtained.
[0214] In an optional implementation manner, the unit for obtaining the operation instruction to be verified obtains the intermediate feature data corresponding to each third feature data by using the following steps:
[0215] According to the arrangement order of the third feature data, select one third feature data from the third feature data sequence as the current third feature data;
[0216] Inputting the current third feature data and the intermediate feature data corresponding to the previous third feature data into the first recurrent neural network to obtain the intermediate feature data corresponding to the current third feature data; wherein the intermediate feature data corresponding to the previous third feature data is obtained based on the previous third feature data and the first recurrent neural network;
[0217] Return to the step of selecting one third feature data from the third feature data sequence as the current third feature data according to the arrangement order of the third feature data, until the intermediate feature data of all the third feature data in the third feature data sequence are obtained.
[0218] In an optional implementation, the detection result includes a third probability value and a third category;
[0219] The unit for obtaining the operation instruction to be verified obtains the operation instruction to be verified corresponding to the intermediate video data by using the following steps:
[0220] Setting a weight value for each third intermediate feature data in a preset manner;
[0221] Based on the third probability value, the third category and the set weight value corresponding to each third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data is determined.
[0222] In an optional implementation, after receiving the original video data sent by the user terminal based on the target operation instruction, before inputting the original video data into a pre-trained neural network model for processing and outputting the identified operation instruction to be verified, the device further includes:
[0223] A judgment module, used to judge whether the duration of the original video data meets the set duration condition, and if so, input the original video data into a pre-trained neural network model;
[0224] If not satisfied, a return signal is sent to the user terminal, so that the user terminal re-sends the updated original video data based on the target operation instruction.
[0225] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.
[0226] like Figure 2 As shown, an embodiment of the present application also provides an electronic device, including: a processor 220, a storage medium 240 and a communication bus 230, wherein the storage medium stores machine-readable instructions executable by the processor 220, and when the electronic device is running, the processor 220 communicates with the storage medium through the communication bus 230, and the processor 220 executes the machine-readable instructions to perform the steps of the vehicle verification method provided in the embodiment of the present application.
[0227] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the vehicle verification method provided in the embodiment of the present application are executed.
[0228] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0229] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0230] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0231] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.
[0232] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A vehicle verification method, characterized in that: The method comprises: In response to a vehicle verification request sent by a user terminal, a target operation instruction is selected from a pre-stored verification instruction set, and the target operation instruction is sent to the user terminal; the verification instruction set includes a plurality of operation instructions for instructing to perform different operations on the vehicle; Receiving original video data sent by the user terminal based on the target operation instruction, inputting the original video data into a pre-trained neural network model for processing, and outputting the identified operation instruction to be verified; Determining a vehicle legality verification result corresponding to the vehicle verification request based on a matching result between the operation instruction to be verified and the target operation instruction; The pre-trained neural network model includes a target detection model and an instruction recognition model, and the raw video data is input into the pre-trained neural network model for processing, and the recognized operation instruction to be verified is output, including: Input the original video data into the trained target detection model to obtain the target component area in each frame image corresponding to the original video data; the target component area is the area where the effective components of the different operations performed on the vehicle are located; Based on the target component region in each acquired frame of image, generating intermediate video data containing the target component region; The intermediate video data is input into the instruction recognition model to obtain the recognized operation instruction to be verified.
2. The method according to claim 1, characterized in that The verification instruction set includes a plurality of operation instructions for instructing to perform different operations on the vehicle lights.
3. The method according to claim 2, characterized in that The verification instruction set includes multiple of the following operation instructions: Turn on the left flash, turn on the right flash, turn on both flashes, turn on the headlights, turn on the headlights and turn on the left flash, turn on the headlights and turn on the right flash, turn on the headlights and turn on both flashes.
4. The method according to claim 1, characterized in that: The target detection model includes a first convolutional neural network, a region recommendation network, a region mapping pooling layer, a fully connected network, a regression network and a first classifier; The step of inputting the original video data into a trained target detection model to obtain a target component region in each frame image corresponding to the original video data includes: For each frame of image corresponding to the original video data, the following process is performed: Inputting the image into the first convolutional neural network for convolution processing to obtain first feature data of the image, wherein the first feature data is used to characterize local features of the image; Inputting the first feature data into the region recommendation network and the region mapping pooling layer, processing the first feature data through the region recommendation network, and obtaining coordinate information of the candidate target region corresponding to the first feature data; inputting the coordinate information of the candidate target region into the region mapping pooling layer, mapping the coordinate information of the candidate target region onto the first feature data through the region mapping pooling layer, and obtaining second feature data corresponding to the candidate target region from the first feature data, wherein the second feature data is used to characterize local features of the candidate target region; For each candidate region, the second feature data corresponding to the candidate target region is input into the fully connected network for processing, and the fully connected feature data corresponding to the second feature data is obtained, and the fully connected feature data is input into the regression network and the first classifier, and the fully connected feature data is processed by the regression network to generate regression data corresponding to the candidate target region, and the coordinate information of the candidate target region is changed based on the regression data; the fully connected feature data is processed by the first classifier to obtain the category and probability value of the candidate target region; Based on the changed coordinate information of each candidate target region and the category and probability value corresponding to the candidate target region, the target component region corresponding to the image is obtained.
5. The method according to claim 4, characterized in that The first convolutional neural network includes at least one convolutional layer; Inputting the image into the first convolutional neural network for convolution processing to obtain first feature data of the image includes: The image is convolved by each convolution layer in the at least one convolution layer to obtain first feature data of the image.
6. The method according to claim 4, characterized in that The region recommendation network includes a second convolutional neural network, a target region regression network, and a target region classification network; wherein n types of detection frames are provided in the region recommendation network, wherein the ratios of width to height of different detection frames are different, or the areas of different detection frames are different, and n is a positive integer; The processing of the first feature data by a region recommendation network to obtain location information of a candidate target region corresponding to the first feature data of the image includes: Inputting the first feature data into the second convolutional neural network for convolution processing to obtain first intermediate feature data corresponding to the first feature data, wherein the first intermediate feature data includes a plurality of first intermediate feature vectors, each of which corresponds to initial coordinate information of n types of detection boxes; The first intermediate feature data are respectively input into the target area regression network and the target area classification network, and the first intermediate feature data are processed by the target area regression network to obtain second regression data of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data, and the target coordinate information of the detection frame is obtained based on the second regression data and the initial coordinate information of the detection frame; the first intermediate feature data are processed by the target area classification network to obtain second probability values and second categories of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data, and the position information of the candidate target area corresponding to the first feature data is obtained based on the second probability values of multiple detection frames, the second category and the target coordinate information of each detection frame.
7. The method according to claim 1, characterized in that The instruction recognition model includes: a third convolutional neural network, a first recurrent neural network, and a second classifier; The step of inputting the intermediate video data into the instruction recognition model to obtain the recognized operation instruction to be verified includes: Inputting the intermediate video data into the third convolutional neural network for convolution processing to obtain a third feature data sequence, wherein the third feature data sequence includes third feature data corresponding to each frame image in the intermediate video data, and an arrangement order of the third feature data in the third feature data sequence is the same as an arrangement order of each frame image in the intermediate video data; Inputting the third feature data sequence into the first recurrent neural network, obtaining third intermediate feature data corresponding to each of the third feature data, wherein the third intermediate feature data is used to characterize the local features of any frame image in the intermediate video data; Input each third intermediate feature data into the second classifier to obtain a detection result corresponding to each third intermediate feature data; Based on the detection result corresponding to each third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data is obtained.
8. The method according to claim 7, characterized in that The step of inputting the third feature data sequence into the first recurrent neural network to obtain intermediate feature data corresponding to each of the third feature data comprises: According to the arrangement order of the third feature data, selecting a third feature data from the third feature data sequence as the current third feature data; Inputting the current third feature data and the intermediate feature data corresponding to the previous third feature data into the first recurrent neural network to obtain the intermediate feature data corresponding to the current third feature data; wherein the intermediate feature data corresponding to the previous third feature data is obtained based on the previous third feature data and the first recurrent neural network; Return to the step of selecting one third feature data from the third feature data sequence as the current third feature data according to the arrangement order of the third feature data, until the intermediate feature data of all the third feature data in the third feature data sequence is obtained.
9. The method according to claim 7, characterized in that: The detection result includes a third probability value and a third category; The obtaining, based on the detection result corresponding to each third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data comprises: Setting a weight value for each third intermediate feature data in a preset manner; Based on the third probability value, the third category and the set weight value corresponding to each of the third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data is determined.
10. The method according to claim 1, characterized in that After receiving the original video data sent by the user terminal based on the target operation instruction, before inputting the original video data into a pre-trained neural network model for processing and outputting the identified operation instruction to be verified, the method further includes: Determine whether the duration of the original video data meets the set duration condition, and if so, input the original video data into a pre-trained neural network model; If not satisfied, a return signal is sent to the user terminal, so that the user terminal re-sends the updated original video data based on the target operation instruction.
11. A vehicle verification device, characterized in that: The device comprises: A response module, used to respond to the vehicle verification request sent by the user terminal, select a target operation instruction from a pre-stored verification instruction set, and send the target operation instruction to the user terminal; the verification instruction set includes a plurality of operation instructions for instructing to perform different operations on the vehicle; A recognition module, used for receiving the original video data sent by the user terminal based on the target operation instruction, inputting the original video data into a pre-trained neural network model for processing, and outputting the recognized operation instruction to be verified; A determination module, configured to determine a vehicle legality verification result corresponding to the vehicle verification request based on a matching result between the operation instruction to be verified and the target operation instruction; The pre-trained neural network model includes a target detection model and an instruction recognition model. When identifying the operation instruction to be verified corresponding to the original video data, the recognition module includes: A target component region acquisition unit is used to input the original video data into a trained target detection model to acquire a target component region in each frame image corresponding to the original video data; the target component region is a region where the effective components of the different operations performed on the vehicle are located; An intermediate video data generating unit, configured to generate intermediate video data containing the target component region based on the target component region in each acquired frame of image; The unit for acquiring the operation instruction to be verified is used to input the intermediate video data into the instruction recognition model to acquire the recognized operation instruction to be verified.
12. The device according to claim 11, characterized in that The verification instruction set includes a plurality of operation instructions for instructing to perform different operations on the vehicle lights.
13. The device according to claim 12, characterized in that The verification instruction set includes multiple of the following operation instructions: Turn on the left flash, turn on the right flash, turn on both flashes, turn on the headlights, turn on the headlights and turn on the left flash, turn on the headlights and turn on the right flash, turn on the headlights and turn on both flashes.
14. The device according to claim 11, characterized in that The target detection model includes a first convolutional neural network, a region recommendation network, a region mapping pooling layer, a fully connected network, a regression network and a first classifier; The target component region acquisition unit acquires the target component region in each frame image corresponding to the original video data by using the following steps: For each frame of image corresponding to the original video data, the following process is performed: Inputting the image into the first convolutional neural network for convolution processing to obtain first feature data of the image, wherein the first feature data is used to characterize local features of the image; Inputting the first feature data into the region recommendation network and the region mapping pooling layer, processing the first feature data through the region recommendation network, and obtaining coordinate information of the candidate target region corresponding to the first feature data; inputting the coordinate information of the candidate target region into the region mapping pooling layer, mapping the coordinate information of the candidate target region onto the first feature data through the region mapping pooling layer, and obtaining second feature data corresponding to the candidate target region from the first feature data, wherein the second feature data is used to characterize local features of the candidate target region; For each candidate region, the second feature data corresponding to the candidate target region is input into the fully connected network for processing, and the fully connected feature data corresponding to the second feature data is obtained, and the fully connected feature data is input into the regression network and the first classifier, and the fully connected feature data is processed by the regression network to generate regression data corresponding to the candidate target region, and the coordinate information of the candidate target region is changed based on the regression data; the fully connected feature data is processed by the first classifier to obtain the category and probability value of the candidate target region; Based on the changed coordinate information of each candidate target region and the category and probability value corresponding to the candidate target region, the target component region corresponding to the image is obtained.
15. The device according to claim 14, characterized in that The first convolutional neural network includes at least one convolutional layer; The target component region acquisition unit acquires the first feature data of the image by using the following steps: The image is convolved by each convolution layer in the at least one convolution layer to obtain first feature data of the image.
16. The device according to claim 14, characterized in that The region recommendation network includes a second convolutional neural network, a target region regression network, and a target region classification network; wherein n types of detection frames are provided in the region recommendation network, wherein the ratios of width to height of different detection frames are different, or the areas of different detection frames are different, and n is a positive integer; The target component region acquisition unit acquires the position information of the candidate target region corresponding to the first feature data of the image by using the following steps: Inputting the first feature data into the second convolutional neural network for convolution processing to obtain first intermediate feature data corresponding to the first feature data, wherein the first intermediate feature data includes a plurality of first intermediate feature vectors, each of which corresponds to initial coordinate information of n types of detection boxes; The first intermediate feature data are respectively input into the target area regression network and the target area classification network, and the first intermediate feature data are processed by the target area regression network to obtain second regression data of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data, and the target coordinate information of the detection frame is obtained based on the second regression data and the initial coordinate information of the detection frame; the first intermediate feature data are processed by the target area classification network to obtain second probability values and second categories of each of the n detection frames corresponding to each first intermediate feature vector in the first intermediate feature data, and the position information of the candidate target area corresponding to the first feature data is obtained based on the second probability values of multiple detection frames, the second category and the target coordinate information of each detection frame.
17. The device according to claim 11, characterized in that The instruction recognition model includes: a third convolutional neural network, a first recurrent neural network, and a second classifier; The unit for acquiring the operation instruction to be verified acquires the identified operation instruction to be verified by using the following steps: Inputting the intermediate video data into the third convolutional neural network for convolution processing to obtain a third feature data sequence, wherein the third feature data sequence includes third feature data corresponding to each frame image in the intermediate video data, and an arrangement order of the third feature data in the third feature data sequence is the same as an arrangement order of each frame image in the intermediate video data; Inputting the third feature data sequence into the first recurrent neural network, obtaining third intermediate feature data corresponding to each of the third feature data, wherein the third intermediate feature data is used to characterize the local features of any frame image in the intermediate video data; Input each third intermediate feature data into the second classifier to obtain a detection result corresponding to each third intermediate feature data; Based on the detection result corresponding to each third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data is obtained.
18. The device according to claim 17, characterized in that The to-be-verified operation instruction acquisition unit acquires the intermediate feature data corresponding to each of the third feature data by using the following steps: According to the arrangement order of the third feature data, selecting a third feature data from the third feature data sequence as the current third feature data; Inputting the current third feature data and the intermediate feature data corresponding to the previous third feature data into the first recurrent neural network to obtain the intermediate feature data corresponding to the current third feature data; wherein the intermediate feature data corresponding to the previous third feature data is obtained based on the previous third feature data and the first recurrent neural network; Return to the step of selecting one third feature data from the third feature data sequence as the current third feature data according to the arrangement order of the third feature data, until the intermediate feature data of all the third feature data in the third feature data sequence is obtained.
19. The device according to claim 17, characterized in that The detection result includes a third probability value and a third category; The to-be-verified operation instruction acquisition unit obtains the to-be-verified operation instruction corresponding to the intermediate video data by using the following steps: Setting a weight value for each third intermediate feature data in a preset manner; Based on the third probability value, the third category and the set weight value corresponding to each of the third intermediate feature data, the operation instruction to be verified corresponding to the intermediate video data is determined.
20. The device according to claim 11, characterized in that After receiving the original video data sent by the user terminal based on the target operation instruction, before inputting the original video data into a pre-trained neural network model for processing and outputting the identified operation instruction to be verified, the device further includes: A judgment module, used to judge whether the duration of the original video data meets the set duration condition, and if so, input the original video data into a pre-trained neural network model; If not satisfied, a return signal is sent to the user terminal, so that the user terminal re-sends the updated original video data based on the target operation instruction.
21. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the vehicle verification method as described in any one of claims 1 to 10.
22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the vehicle verification method according to any one of claims 1 to 10 are executed.
