Vehicle damage assessment method and device
By combining voice recognition and image recognition technologies, the system automatically completes the damage assessment and labeling of vehicle damage information, solving the problem of low efficiency in existing technologies and achieving a highly efficient and accurate vehicle damage assessment process.
Patent Information
- Application Number
- CN202110112020.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-27
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-01-27
AI Technical Summary
In the existing vehicle damage detection model, vehicle inspectors need to take a large number of vehicle defect images and manually mark them, resulting in low efficiency and long time consumption.
A method combining speech recognition and image recognition is adopted. By acquiring vehicle damage video information, a pre-trained speech recognition model is used to perform speech recognition, extract vehicle damage video segments, and an image recognition model is used to perform image recognition, generating a damage assessment labeling document.
It enables automatic damage assessment annotation without requiring users to manually annotate vehicle damage images, improving the efficiency and accuracy of damage assessment annotation document generation.
Smart Images

Figure CN112818811B_ABST
Abstract
Description
Technical Field
[0001] This document relates to the field of artificial intelligence, and in particular to a vehicle damage assessment method and device. Background Art
[0002] At present, with the advent of the Internet era, the Internet has been widely used in people's daily study, work and life. People's daily affairs can be processed and presented through the Internet. At the same time, with the rapid development of mobile Internet, online second-hand car transactions and online car rentals have become widely popular and favored by most users.
[0003] Among them, used car trading services and online car rental services both require vehicle damage assessment and labeling. Currently, under the existing vehicle damage detection model, vehicle inspectors usually need to take a large number of vehicle defect images and manually mark the defect information, which is time-consuming and inefficient. Therefore, it is necessary to provide a vehicle damage assessment technical solution with high efficiency and accuracy. Summary of the invention
[0004] The purpose of one or more embodiments of this specification is to provide a vehicle damage assessment method. The vehicle damage assessment method includes:
[0005] Acquire vehicle damage video information of the target vehicle, wherein the vehicle damage video information includes: damage description voice information for damaged parts of the vehicle and video frames of damaged areas;
[0006] Using a pre-trained speech recognition model, speech recognition is performed on the damage description speech information to obtain a damage speech recognition result of the target vehicle; wherein the damage speech recognition result includes: time stamp information, damaged component identification and component damage type;
[0007] extracting at least one vehicle damage video segment from the vehicle damage video information based on the damage speech recognition result;
[0008] Using a pre-trained image recognition model, image recognition is performed on the damaged area video frame in the vehicle damage video clip to obtain a damaged image recognition result of the target vehicle;
[0009] Based on the damage speech recognition result and the damage image recognition result, damage assessment and annotation document information of the target vehicle is generated.
[0010] The purpose of one or more embodiments of this specification is to provide a vehicle damage assessment device. The vehicle damage assessment device includes:
[0011] A video information acquisition module, used to acquire vehicle damage video information of a target vehicle, wherein the vehicle damage video information includes: damage description voice information for damaged vehicle parts and a video frame of a damaged area;
[0012] A speech recognition module, used to perform speech recognition on the damage description speech information using a pre-trained speech recognition model to obtain a damage speech recognition result of the target vehicle; wherein the damage speech recognition result includes: time stamp information, damaged component identification and component damage type;
[0013] A video segment extraction module, configured to extract at least one vehicle damage video segment from the vehicle damage video information based on the damage speech recognition result;
[0014] An image recognition module is used to perform image recognition on the damaged area video frame in the vehicle damage video clip using a pre-trained image recognition model to obtain a damaged image recognition result of the target vehicle;
[0015] The vehicle damage assessment module is used to generate damage assessment annotation document information of the target vehicle based on the damage voice recognition result and the damage image recognition result.
[0016] One or more embodiments of the present specification aim to provide a vehicle damage assessment device, comprising: a processor; and a memory arranged to store computer executable instructions.
[0017] When the computer executable instructions are executed, the processor is caused to obtain vehicle damage video information of the target vehicle, wherein the vehicle damage video information includes: damage description voice information for damaged parts of the vehicle and a video frame of the damaged area;
[0018] Using a pre-trained speech recognition model, speech recognition is performed on the damage description speech information to obtain a damage speech recognition result of the target vehicle; wherein the damage speech recognition result includes: time stamp information, damaged component identification and component damage type;
[0019] extracting at least one vehicle damage video segment from the vehicle damage video information based on the damage speech recognition result;
[0020] Using a pre-trained image recognition model, image recognition is performed on the damaged area video frame in the vehicle damage video clip to obtain a damaged image recognition result of the target vehicle;
[0021] Based on the damage speech recognition result and the damage image recognition result, damage assessment and annotation document information of the target vehicle is generated.
[0022] The purpose of one or more embodiments of the present specification is to provide a storage medium for storing computer executable instructions. When the executable instructions are executed by a processor, the vehicle damage video information of the target vehicle is obtained, wherein the vehicle damage video information includes: damage description voice information for the damaged parts of the vehicle and a video frame of the damaged area;
[0023] Using a pre-trained speech recognition model, speech recognition is performed on the damage description speech information to obtain a damage speech recognition result of the target vehicle; wherein the damage speech recognition result includes: time stamp information, damaged component identification and component damage type;
[0024] extracting at least one vehicle damage video segment from the vehicle damage video information based on the damage speech recognition result;
[0025] Using a pre-trained image recognition model, image recognition is performed on the damaged area video frame in the vehicle damage video clip to obtain a damaged image recognition result of the target vehicle;
[0026] Based on the damage speech recognition result and the damage image recognition result, damage assessment and annotation document information of the target vehicle is generated.
[0027] In the vehicle damage assessment method and device provided by one or more embodiments of the present specification, after obtaining the vehicle damage video information of the target vehicle, voice recognition is first performed on the vehicle damage video information, and a vehicle damage video segment is extracted from the vehicle damage video information. Then, image recognition is performed on the vehicle damage video segment to obtain a damage image recognition result, so that the required vehicle damage video frame can be extracted from the damage video segment based on the damage image recognition result. Finally, the vehicle damage video frame is annotated for damage assessment based on the damage voice recognition result and the damage image recognition result, thereby realizing automatic completion of damage assessment and annotation of vehicle damage information based on the vehicle damage video information by combining voice recognition and image recognition, without the need for the user to manually annotate the vehicle damage image and create a vehicle damage assessment annotation document, thereby improving the generation efficiency and accuracy of the damage assessment annotation document of the target vehicle. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0029] Figure 1A schematic diagram of an application scenario of a vehicle damage assessment system provided in one or more embodiments of this specification;
[0030] Figure 2 A first flow chart of a vehicle damage assessment method provided in one or more embodiments of this specification;
[0031] Figure 3 A second flow chart of a vehicle damage assessment method provided in one or more embodiments of this specification;
[0032] Figure 4 A third flow chart of a vehicle damage assessment method provided in one or more embodiments of this specification;
[0033] Figure 5 A fourth flow chart of a vehicle damage assessment method provided in one or more embodiments of this specification;
[0034] Figure 6 A schematic diagram of the module composition of a vehicle damage assessment device provided in one or more embodiments of this specification;
[0035] Figure 7 A schematic diagram of the structure of a vehicle damage assessment device provided in one or more embodiments of this specification. DETAILED DESCRIPTION
[0036] In order to enable persons skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only one or more partial embodiments of this specification, rather than all embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.
[0037] It should be noted that, in the absence of conflict, one or more embodiments and features in the embodiments in this specification can be combined with each other. One or more embodiments of this specification will be described in detail below with reference to the drawings and in combination with the embodiments.
[0038] One or more embodiments of the present specification provide a vehicle damage assessment method and device. After obtaining vehicle damage video information of a target vehicle, voice recognition is first performed on the vehicle damage video information, a vehicle damage video segment is extracted from the vehicle damage video information, and then image recognition is performed on the vehicle damage video segment to obtain a damage image recognition result, so as to extract the required vehicle damage video frame from the damage video segment based on the damage image recognition result. Finally, damage assessment is performed on the vehicle damage video frame based on the damage voice recognition result and the damage image recognition result, thereby realizing automatic completion of damage assessment and annotation of vehicle damage information based on vehicle damage video information by combining voice recognition and image recognition, without the need for users to manually annotate the vehicle damage image and create a vehicle damage assessment annotation document, thereby improving the generation efficiency and accuracy of the damage assessment annotation document of the target vehicle.
[0039] Figure 1 A schematic diagram of an application scenario of a vehicle damage assessment system provided in one or more embodiments of the present specification, the system comprising: a client and / or a backend server, wherein in the case where a pre-trained model is deployed on the client, the client uses the pre-trained model to identify and analyze the collected vehicle damage video information to obtain damage assessment annotation document information; or, in the case where a pre-trained model is deployed on the server, the client collects vehicle damage video information and uploads it to the backend server, and the backend server uses the pre-trained model to identify and analyze the collected vehicle damage video information to obtain damage assessment annotation document information; Figure 1 As shown, a vehicle damage assessment system including a client and a backend server is taken as an example, wherein the client may be a mobile terminal such as a smart phone or a tablet computer, the client may also be a terminal device such as a personal computer, and the backend server may be an independent server or a server cluster composed of multiple servers. Specifically, the specific process of vehicle damage assessment is as follows:
[0040] The client collects vehicle damage video information of the target vehicle through a camera device, wherein the vehicle damage video information may include: damage description voice information of the damaged parts of the vehicle and a video frame of the damaged area;
[0041] The client uploads the collected vehicle damage video information to the backend server;
[0042] After receiving the vehicle damage video information, the backend server uses a pre-trained speech recognition model to perform speech recognition on the damage description speech information in the vehicle damage video information to obtain a damage speech recognition result of the target vehicle; wherein the damage speech recognition result may include: time stamp information, damaged component identification and component damage type;
[0043] The backend server extracts at least one vehicle damage video clip from the vehicle damage video information based on the time stamp information in the damage speech recognition result;
[0044] The backend server uses a pre-trained image recognition model to perform image recognition on the damaged area video frame in the above vehicle damage video clip to obtain a damaged image recognition result of the target vehicle; wherein the damaged image recognition result may include: component damage detection result and / or image clarity detection result, and the component damage detection result may include: damage location information and component damage type;
[0045] The backend server generates damage assessment and annotation document information of the target vehicle based on the above-mentioned damage voice recognition results and damage image recognition results, wherein the damage assessment and annotation document information may include: multiple damage assessment and annotation records, each damage assessment and annotation record includes: at least one of the vehicle damage area image, damaged component identification, damage location information and component damage type.
[0046] In the above application scenario, after obtaining the vehicle damage video information of the target vehicle, voice recognition is first performed on the vehicle damage video information, and a vehicle damage video clip is extracted from the vehicle damage video information. Then, image recognition is performed on the vehicle damage video clip to obtain a damage image recognition result, so as to extract the required vehicle damage video frame from the damage video clip based on the damage image recognition result. Finally, the vehicle damage video frame is marked for damage assessment based on the damage voice recognition result and the damage image recognition result. In this way, the damage assessment and annotation of the vehicle damage information is automatically completed based on the vehicle damage video information by combining voice recognition and image recognition. There is no need for users to manually mark the damage on the vehicle damage image and prepare a vehicle damage assessment and annotation document, thereby improving the generation efficiency and accuracy of the damage assessment and annotation document of the target vehicle.
[0047] Figure 2 A first flow chart of a vehicle damage assessment method provided in one or more embodiments of this specification, Figure 2 The method in can be Figure 1 The client or backend server in the Figure 2 As shown, the method comprises at least the following steps:
[0048] S202, obtaining vehicle damage video information of the target vehicle, wherein the vehicle damage video information includes: damage description voice information for damaged parts of the vehicle and a video frame of the damaged area;
[0049] Specifically, under the control of the vehicle damage assessor, the client records vehicle damage video information for all damaged vehicle parts through a camera device. Correspondingly, the damage description voice information is the damage introduction voice of the vehicle damage assessor for the damaged vehicle part currently being recorded, for example, XX part (part identification) has YY damage (damage type), and the damage area video frame is the damage area image of the vehicle damage assessor for the damaged vehicle part currently being recorded.
[0050] S204, using a pre-trained speech recognition model, performing speech recognition on the damage description speech information to obtain a damage speech recognition result of the target vehicle; wherein the damage speech recognition result includes: time stamp information, damaged component identification and component damage type;
[0051] Specifically, after obtaining the vehicle damage video information, feature extraction is performed on the vehicle damage video information to obtain a damage video feature vector, and the damage video feature vector is input into a pre-trained speech recognition model. The output of the speech recognition model is the damage speech recognition result corresponding to each vehicle damaged component of the target vehicle;
[0052] For example, components 1, 2, and 3 of the target vehicle are damaged to a certain extent, and the vehicle damage video information includes: damage description voice information 1 and damaged area video frame 1 of vehicle component 1, damage description voice information 2 and damaged area video frame 2 of vehicle component 2, and damage description voice information 3 and damaged area video frame 3 of vehicle component 3. Correspondingly, based on the damage description voice information 1, the above-mentioned voice recognition model is used to obtain a damage voice recognition result 1, wherein the time mark information in the damage voice recognition result 1 is the playback time node 1 (such as 2′35″) of the damage description voice information 1 and the damaged area video frame 1 in the vehicle damage video information. Similarly, based on the damage description voice information 2, the above-mentioned voice recognition model is used to obtain a damage voice recognition result 2; based on the damage description voice information 3, the above-mentioned voice recognition model is used to obtain a damage voice recognition result 3. That is, in the case where the target vehicle includes multiple damages, the damage voice recognition result of the target vehicle includes multiple voice recognition results, each voice recognition result corresponds to a damage area, and each voice recognition result includes time mark information, a damaged component identifier, and a component damage type.
[0053] S206, extracting at least one vehicle damage video segment from the vehicle damage video information based on the damage speech recognition result;
[0054] Specifically, based on the time stamp information in the damage speech recognition result, the playback time node (i.e., the time axis position in the video) of the damage description voice information and the damage area video frame recorded for each damaged vehicle component can be located, and then for each time stamp information, the corresponding vehicle damage video clips are extracted from the vehicle damage video information respectively, and each vehicle damage video clip contains the damage description voice information and the damage area video frame recorded for one vehicle damage; wherein, for the case where the target vehicle includes multiple damages, the number of extracted vehicle damage video clips is multiple, and each damage corresponds to a vehicle damage video clip; still taking the case where components 1, 2, and 3 of the target vehicle have a certain degree of damage as an example, vehicle damage video clip 1 corresponding to damaged component 1, vehicle damage video clip 2 corresponding to damaged component 2, and vehicle damage video clip 3 corresponding to damaged component 3 are extracted.
[0055] S208, using a pre-trained image recognition model, performing image recognition on the damaged area video frame in the vehicle damage video clip to obtain a damaged image recognition result of the target vehicle;
[0056] Among them, the above-mentioned damage image recognition results include: component damage detection results, or component damage detection results and image clarity detection results. For the case where the damage image recognition results include component damage detection results and image clarity detection results, the above-mentioned image recognition model may include two independent neural network models, one for vehicle component damage detection and the other for image clarity detection; it may also include a multi-task neural network model obtained based on multi-task learning, and the loss function corresponding to the neural network model is determined according to the first loss function corresponding to the vehicle component damage detection task and the second loss function corresponding to the image clarity detection task.
[0057] Specifically, for each vehicle damage video clip, feature extraction is performed on the damaged area video frames (which may be all video frames or key video frames) in the vehicle damage video clip to obtain multiple video frame feature vectors, and the video frame feature vectors are input into a pre-trained image recognition model. The output of the image recognition model is the damage image recognition results corresponding to each damaged vehicle component of the target vehicle; the image recognition results obtained for each vehicle damage video clip include multiple image recognition results, each image recognition result corresponds to a damaged area video frame recorded for a damaged component, and each image recognition result includes at least one of the damage position information of the damaged area in the damaged area video frame, the component damage type, and the image clarity detection result.
[0058] S210, generating damage assessment and annotation document information of the target vehicle based on the above damage speech recognition results and damage image recognition results.
[0059] Specifically, after extracting vehicle damage video clips from vehicle damage video information using voice recognition and obtaining image recognition results based on the vehicle damage video clips using image recognition, required vehicle damage video frames can be extracted from each vehicle damage video clip based on the image recognition results, and damage assessment annotations can be performed on the vehicle damage video frames based on the damaged component identification and component damage type in the voice recognition results, and the damage location information and component damage type in the damage image recognition results to obtain annotated vehicle damage area images, and damage assessment annotation document information of the target vehicle can be generated based on multiple annotated vehicle damage area images;
[0060] Wherein, in case that the target vehicle includes multiple damages, the above-mentioned damage assessment annotation document information may include: multiple damage assessment annotation records, each damage assessment annotation record corresponds to one vehicle damage, and each damage assessment annotation record includes: at least one of: a vehicle damage area image, a damaged component identification, damage location information and a component damage type; specifically, the component damage type may be determined based on a damage voice recognition result, or based on a damage image recognition result, or based on a damage voice recognition result and a damage image recognition result. If the component damage types in the damage voice recognition result and the damage image recognition result are inconsistent, the component damage type with a high confidence is determined as the final component damage type, that is, if the confidence of the component damage type in the damage voice recognition result (the confidence is determined when the component damage type is obtained using a voice recognition model) is greater than the confidence of the component damage type in the damage image recognition result (the confidence is determined when the component damage type is obtained using an image recognition model), the component damage type in the damage voice recognition result is determined as the final component damage type, otherwise, the component damage type in the damage image recognition result is determined as the final component damage type.
[0061] Among them, due to the combination of voice recognition and image recognition, the target vehicle is marked for damage assessment. In the case that the shape, area, color and other feature differences of the vehicle damage are large within the class or small between classes, the voice recognition results can ensure the accuracy of the vehicle damage assessment marking. Therefore, the final damage assessment marking document information will not be affected by the limited effect of image recognition in certain special cases.
[0062] In one or more embodiments of the present specification, after obtaining the vehicle damage video information of the target vehicle, voice recognition is first performed on the vehicle damage video information, a vehicle damage video clip is extracted from the vehicle damage video information, and then image recognition is performed on the vehicle damage video clip to obtain a damage image recognition result, so that the required vehicle damage video frame can be extracted from the damage video clip based on the damage image recognition result, and finally, the vehicle damage video frame is annotated for damage assessment based on the damage voice recognition result and the damage image recognition result, thereby realizing the automatic completion of the damage assessment and annotation of the vehicle damage information based on the vehicle damage video information by combining voice recognition and image recognition, without the need for the user to manually annotate the damage on the vehicle damage image and prepare a vehicle damage assessment and annotation document, thereby improving the generation efficiency and accuracy of the damage assessment and annotation document of the target vehicle.
[0063] Furthermore, in order to improve the accuracy of the damage image recognition results, and thus improve the accuracy of the annotation of the damage image, the image recognition process includes not only damage area detection but also image clarity detection, and the image recognition model is pre-trained based on the two tasks of damage area detection and image clarity detection. Based on this, the above-mentioned image recognition model includes: a multi-task neural network model obtained based on multi-task learning; wherein the loss function corresponding to the multi-task neural network model is obtained by weighted calculation based on a first loss function corresponding to the component damage detection task and a second loss function corresponding to the image clarity detection task;
[0064] Correspondingly, such as Figure 3 As shown, the above S208 uses a pre-trained image recognition model to perform image recognition on the damaged area video frame in the above vehicle damage video clip to obtain the damaged image recognition result of the target vehicle, which specifically includes:
[0065] S2082, extracting key video frames from the damaged area video frames in the vehicle damage video clip as video frame images to be identified;
[0066] Specifically, each damaged area video frame in the vehicle damage video clip can be used as the video frame image to be identified, or key video frames in the vehicle damage video clip can be used as the video frame image to be identified through key frame extraction.
[0067] S2084, using a pre-trained multi-task neural network model, performing component damage detection and image clarity detection on the video frame image to be identified, and obtaining a component damage detection result and an image clarity detection result of the target vehicle; wherein the component damage detection result includes: component damage type and damage location information;
[0068] Specifically, for each video frame image to be identified in the currently identified vehicle damage video clip, feature extraction is performed on the video frame image to be identified to obtain a corresponding video frame feature vector (i.e., a damage image feature vector), and the video frame feature vector is input into a pre-trained multi-task neural network model. The output of the multi-task neural network model is the component damage detection result and the image clarity detection result of the video frame image to be identified; wherein the above-mentioned component damage detection result includes the component damage type and damage location information corresponding to each video frame image to be identified, and the above-mentioned image clarity detection result includes the image clarity corresponding to each video frame image to be identified.
[0069] S2086: Determine the combination of the above component damage detection results and image clarity detection results as the damage image recognition result of the target vehicle.
[0070] In one or more embodiments of the present specification, an image recognition model is pre-constructed based on a multi-task learning architecture, and model parameters corresponding to two related detection tasks are iteratively trained simultaneously during the model training process. Since the loss function corresponding to the image recognition model is jointly determined by the loss function corresponding to the component damage detection task and the loss function corresponding to the image clarity detection task, the two related detection tasks share feature information and complement each other during the model training process, thereby improving the prediction effect of each detection task in the trained model; and component damage detection and image clarity detection are simultaneously performed using the pre-trained image recognition model, thereby reducing the amount of memory occupied, improving the processing speed, improving the damage detection efficiency, and further improving the efficiency of video frame selection.
[0071] Wherein, in case that the above-mentioned damage image recognition results include: component damage detection results and image clarity detection results, the required target damage video frame can be extracted from the vehicle damage video clip with reference to both the component damage detection results and the image clarity detection results, and then the vehicle damage assessment and labeling can be performed based on the target damage video frame;
[0072] Correspondingly, such as Figure 4 As shown, the above S210 generates damage assessment annotation document information of the target vehicle based on the above damage voice recognition result and damage image recognition result, specifically including:
[0073] S2102, for each vehicle damage video clip, based on the above-mentioned image clarity detection result and component damage detection result, select a target damage video frame in the vehicle damage video clip;
[0074] Specifically, a damaged image video frame whose damage detection result and image clarity both meet corresponding preset constraint conditions is selected from the vehicle damage video clip as a target damaged video frame.
[0075] S2104, based on the component damage detection result and the damage speech recognition result corresponding to the target damaged video frame, the target damaged video frame is damaged and marked to obtain a marked target damaged video frame; wherein the marked target damaged video frame includes: at least one of the target damaged video frame, a damaged component identifier, a component damage type, and damage location information;
[0076] The damaged parts identification may include: front bumper, rear bumper, left front headlight, right front headlight, front hood, left front door, right front door, left rear door, right rear door, left front fender, right front fender, left rear fender, right rear fender, rear hood, front windshield, rear windshield, left rearview mirror, right rearview mirror, left taillight, right taillight, left fog light, right fog light, right front window, left front window, right rear window, left rear window, right bottom edge, left bottom edge, right front door handle, left front door handle, left rear door handle, right rear door handle, left front tire, left rear At least one of the following: tire, right front tire, right rear tire, right front wheel eyebrow, left front wheel eyebrow, right rear wheel eyebrow, left rear wheel eyebrow, left front steel rim, left rear steel rim, right front steel rim, right rear steel rim, left front door trim, right front door trim, left rear door trim, right rear door trim, left front door trim, front bumper trim, rear bumper trim, front bumper guard, rear bumper guard, front bumper deflector, rear bumper deflector, roof, steering wheel, left front seat, right front seat, left rear seat, right rear seat, center console, engine, water tank, upper engine hood, lower engine hood;
[0077] Correspondingly, the component damage type may include at least one of scratches, breakage, rust, scrapes, deformation, paint peeling, cracking, and oil leakage; the damage location information may include the coordinates of at least two boundary points of the damage area, for example, the coordinates of the upper left corner and the lower right corner of the circumscribed rectangular box of the area occupied by the damage location in the target damage video frame (Bbox x1 , Bbox y1 ,Bbox x2 ,Bbox y2 ) to indicate that (Bbox x1 , Bbox y1 ) represents the coordinates of the upper left corner, (Bbox x2 , Bbox y2 ) represents the coordinates of the lower right corner.
[0078] S2106, generating damage assessment annotation document information of the target vehicle based on the marked target damage video frames corresponding to each vehicle damage video clip.
[0079] Specifically, for the process of extracting the required target damage video frame from the vehicle damage video clip, such as Figure 5As shown, the above S2102, for each vehicle damage video clip, based on the above image clarity detection result and component damage detection result, selects a target damage video frame in the vehicle damage video clip, specifically including:
[0080] S21022, selecting, in descending order of image clarity, damaged video frames with the highest image clarity from the vehicle damage video clips as candidate damaged video frames;
[0081] S21024, determining whether the component damage detection result corresponding to the candidate damage video frame satisfies a preset constraint condition; specifically, the preset constraint condition may be that the damage detection result is not empty, or the confidence level corresponding to the damage detection result is greater than a preset threshold;
[0082] If the judgment result is yes, execute S21026 to determine the candidate damaged video frame as the target damaged video frame; specifically, determine the damaged video frame whose component damage detection result meets the preset constraints and has the highest image clarity as the target damaged video frame.
[0083] If the judgment result is no, then return to execute the above S21022, that is, select the next candidate damaged video frame until the target damaged video frame is determined.
[0084] Furthermore, considering that compared with the vehicle damage image obtained by direct shooting, the damaged area video frame in the vehicle damage video information may have a low resolution problem, in order to ensure the image resolution of the vehicle damage area image used in the final damage assessment and annotation document information, the target damaged video frame is firstly subjected to super-resolution processing, and then the target damaged video frame after super-resolution processing is subjected to damage assessment and annotation. Based on this, the above S2104, based on the component damage detection result and the damage speech recognition result corresponding to the target damaged video frame, performs damage marking on the target damaged video frame to obtain the marked target damaged video frame, specifically includes:
[0085] Using a pre-trained image super-resolution model, super-resolution processing is performed on the target damaged video frame to obtain the target damaged video frame after super-resolution processing;
[0086] Specifically, the model parameters of the initial neural network model are iteratively trained using a deep learning method and based on a public super-resolution training sample set until the loss function converges, and the neural network model obtained based on the corresponding model parameters when the loss function converges is determined as the trained image super-resolution model. Among them, the image super-resolution model can be constructed using CNN or other neural networks, the backbone network can use ResNet-50 (residual network) or other network structures, and the loss function can be a linear combination of MSE (mean square error) loss function and Smooth L1 loss function. After the image super-resolution model is trained based on the super-resolution training sample set, the trained image super-resolution model is deployed on a designated client or background server, so that the trained image super-resolution model can be used to perform super-resolution processing on the target damaged video frame to obtain a high-resolution component damage image.
[0087] Based on the above component damage detection results and damage speech recognition results, damage marking is performed on the target damaged video frame after super-resolution processing to obtain a marked target damaged video frame.
[0088] Specifically, based on the damaged component identification and component damage type in the damaged speech recognition results, and the component damage type and damage location information in the component damage detection results, the target damaged video frame after super-resolution processing is marked for damage, that is, the damaged component identification, component damage type and damage location information are marked in the target damaged video frame after super-resolution processing.
[0089] In the process of extracting the vehicle damage video clip, the above S206, based on the above damage speech recognition result, extracts at least one vehicle damage video clip from the above vehicle damage video information, specifically including:
[0090] For each time mark information in the impaired speech recognition result, determine the start interception time information and the end interception time information according to the time mark information and the preset video interception window;
[0091] In the above vehicle damage video information, the vehicle damage video segment corresponding to the time mark information is extracted with the start interception time information as the starting point and the end interception time information as the end point.
[0092] Specifically, after obtaining the damage speech recognition result based on the speech recognition model, for each vehicle damage defect, a video clip near the time axis position of the vehicle damage defect is intercepted from the vehicle damage video information. If the time tag information (i.e., the time axis position) is T tar , the default video capture window is T w , then the captured vehicle damage video clip is the vehicle damage video information [T tar -T w, T tar +T w ] video clips within the time range, wherein the preset video capture window can be set according to actual conditions, for example, T w Set to 2 seconds or 5 seconds; each damage defect of the target vehicle corresponds to a vehicle damage video clip, as well as the damaged component identification and component damage type obtained based on voice recognition.
[0093] In the training process of the speech recognition model, the pre-trained speech recognition model is trained in the following way:
[0094] Step 1: obtaining a speech training sample set; wherein the speech training sample set includes: a public speech training sample set and a vehicle detection speech training sample set; the vehicle detection speech training sample set includes: a correspondence between a vehicle damage description speech and a speech label (damaged component identification and component damage type);
[0095] Step 2: Use deep learning methods to train the initial neural network model based on the above speech training sample set to obtain a trained speech recognition model.
[0096] Specifically, the model parameters of the initial neural network model are iteratively trained based on the speech training sample set using a deep learning method until the loss function converges, and the neural network model obtained based on the corresponding model parameters when the loss function converges is determined as the trained speech recognition model. The speech recognition model can be constructed using LSTM or other neural networks, and the corresponding loss function can use CTC loss, and the model is pre-trained based on a public speech recognition training data set, and then the model is fine-tuned based on the vehicle detection speech training sample set to obtain the final vehicle damage speech recognition model; after the speech recognition model is obtained by training based on the speech training sample set, the trained speech recognition model is deployed on a designated client or background server, so that the trained speech recognition model can be used to perform speech recognition on the acquired vehicle damage video information in the future to obtain the corresponding damage speech recognition results.
[0097] In the training process of the above image recognition model, the above pre-trained image recognition model is trained in the following way:
[0098] Step A, obtaining an image training sample set; wherein the image training sample set includes a plurality of image training samples, each image training sample includes: a correspondence between sample image information, an actual component damage label and an actual image clarity label; wherein the actual component damage label includes: a component damage type label and a component damage location label, and the actual image clarity label may be an image clarity score, and the score is between 0 and 1;
[0099] Step B, based on the above-mentioned image training sample set, the initial multi-task neural network model obtained based on multi-task learning is trained to obtain a trained multi-task neural network model; wherein the multi-task neural network model is used for component damage detection and image clarity detection, and the total loss function corresponding to the multi-task neural network model is obtained by weighted calculation based on a first loss function corresponding to the component damage detection task and a second loss function corresponding to the image clarity detection task;
[0100] Step C: determining the above-trained multi-task neural network model as a pre-trained image recognition model.
[0101] Specifically, the model parameters of the initial multi-task neural network model are iteratively trained based on the image training sample set using a deep learning method until the total loss function converges, and the multi-task neural network model obtained based on the corresponding model parameters when the total loss function converges is determined as the trained image recognition model. Among them, the image recognition model can be constructed based on a multi-task learning architecture, the backbone neural network of the image recognition model can be constructed using a combination of CNN and LSTM, and the corresponding total loss function can use Smooth L1; after the image recognition model is trained based on the image training sample set, the trained image recognition model is deployed on a designated client or background server, so that the trained image recognition model can be used to perform image recognition on the captured vehicle damage video clips to obtain the corresponding damage image recognition results.
[0102] Specifically, the multi-task neural network model includes a component damage detection task branch and an image clarity detection task branch, the component damage detection task branch corresponds to a first loss function, and the image clarity detection task branch corresponds to a second loss function;
[0103] Correspondingly, the above step B, based on the above image training sample set, performs model training on the initial multi-task neural network model obtained based on multi-task learning to obtain a trained multi-task neural network model, specifically including:
[0104] Step B1, based on the above-mentioned image training sample set and the multi-task neural network model, obtain a predicted component damage label and a predicted image clarity label for each sample image information;
[0105] Step B2, based on the predicted component damage label and the actual component damage label, updating the first loss function; and based on the predicted image clarity label and the actual image clarity label, updating the second loss function;
[0106] Specifically, the first loss function is updated based on the loss label difference between the predicted component damage label and the actual component damage label, so that the loss label difference obtained in the next training becomes smaller and smaller, that is, the predicted component damage label becomes closer and closer to the actual component damage label; the second loss function is updated based on the clarity label difference between the predicted image clarity label and the actual image clarity label, so that the clarity label difference obtained in the next training becomes smaller and smaller, that is, the predicted image clarity label becomes closer and closer to the actual image clarity label.
[0107] Step B3, updating the total loss function of the multi-task neural network model based on the updated first loss function and the second loss function;
[0108] Specifically, if the total loss function is L total , the first loss function is L detection , the second loss function is L sharpness , the first weight corresponding to the first loss function is W detection , the second weight corresponding to the second loss function is W sharpness , the total loss function is L total =W detection *L detection +W sharpness *L sharpness ;
[0109] Step B4, using the back propagation algorithm and based on the updated total loss function, iteratively updating the model parameters of the multi-task neural network model until the total loss function converges;
[0110] Step B5, the multi-task neural network model obtained based on the model parameters corresponding to the convergence of the total loss function is determined as the trained multi-task neural network model; specifically, the model parameters of the trained multi-task neural network model are updated with the values of the model parameters corresponding to the convergence of the total loss function.
[0111] Among them, for the determination process of the total loss function, the above step B3, based on the updated first loss function and the second loss function, updates the total loss function of the multi-task neural network model, specifically including:
[0112] Step B31, based on a preset weight dynamic adjustment formula, determining a first weight corresponding to the first loss function and a second weight corresponding to the second loss function; wherein the first weight is used to characterize the training difficulty of the component damage detection task branch, and the second weight is used to characterize the training difficulty of the image clarity detection task branch;
[0113] Step B32, based on the first loss function and its corresponding first weight and the second loss function and its corresponding second weight, updating the total loss function of the multi-task neural network model;
[0114] The above preset weight dynamic adjustment formula is W = (1-K) γ logK, γ represents the influencing factor parameter, and K represents the prediction accuracy of the task branch.
[0115] Specifically, the influencing factor parameter γ corresponding to the component damage detection task branch and the image clarity detection task branch can be the same or different, and they are empirical values. The prediction accuracy K is the correct prediction probability based on the predicted label and the actual label, that is, the quotient of the number of predicted labels consistent with the actual label and the total number of samples. The prediction accuracy K is inversely proportional to the difficulty of task training. The larger the prediction accuracy K, the smaller the training difficulty of the task branch. Correspondingly, the smaller the weight W assigned to the task branch. The goal of dynamic weight adjustment is to make tasks that are more difficult to train have higher weights, so that the model can be effectively trained for difficult tasks, thereby improving the generalization performance of the model.
[0116] The vehicle damage assessment method in one or more embodiments of the present specification, after acquiring the vehicle damage video information of the target vehicle, first performs voice recognition on the vehicle damage video information, extracts the vehicle damage video segment from the vehicle damage video information, and then performs image recognition on the vehicle damage video segment to obtain a damage image recognition result, so as to extract the required vehicle damage video frame from the damage video segment based on the damage image recognition result, and finally performs damage assessment annotation on the vehicle damage video frame based on the damage voice recognition result and the damage image recognition result, thereby realizing the automatic completion of the damage assessment annotation of the vehicle damage information based on the vehicle damage video information by combining voice recognition and image recognition, without the need for the user to manually annotate the damage on the vehicle damage image and prepare a vehicle damage assessment annotation document, thereby improving the generation efficiency and accuracy of the damage assessment annotation document of the target vehicle.
[0117] Corresponding to the above Figures 2 to 5 The vehicle damage assessment method described above is based on the same technical concept, and one or more embodiments of this specification also provide a vehicle damage assessment device, Figure 6 A schematic diagram of the module composition of a vehicle damage assessment device provided in one or more embodiments of this specification, the device is used to perform Figures 2 to 5 The vehicle damage assessment method described, such as Figure 6 As shown, the device comprises:
[0118] The video information acquisition module 602 is used to acquire vehicle damage video information of the target vehicle, wherein the vehicle damage video information includes: damage description voice information for the damaged parts of the vehicle and video frames of the damaged area;
[0119] The speech recognition module 604 is used to perform speech recognition on the damage description speech information using a pre-trained speech recognition model to obtain a damage speech recognition result of the target vehicle; wherein the damage speech recognition result includes: time stamp information, damaged component identification and component damage type;
[0120] A video segment extraction module 606, configured to extract at least one vehicle damage video segment from the vehicle damage video information based on the damage speech recognition result;
[0121] An image recognition module 608 is used to perform image recognition on the damaged area video frame in the vehicle damage video clip using a pre-trained image recognition model to obtain a damaged image recognition result of the target vehicle;
[0122] The vehicle damage assessment module 610 is used to generate damage assessment annotation document information of the target vehicle based on the damage voice recognition result and the damage image recognition result.
[0123] In one or more embodiments of the present specification, after obtaining the vehicle damage video information of the target vehicle, voice recognition is first performed on the vehicle damage video information, a vehicle damage video clip is extracted from the vehicle damage video information, and then image recognition is performed on the vehicle damage video clip to obtain a damage image recognition result, so that the required vehicle damage video frame can be extracted from the damage video clip based on the damage image recognition result, and finally, the vehicle damage video frame is annotated for damage assessment based on the damage voice recognition result and the damage image recognition result, thereby realizing the automatic completion of the damage assessment and annotation of the vehicle damage information based on the vehicle damage video information by combining voice recognition and image recognition, without the need for the user to manually annotate the damage on the vehicle damage image and prepare a vehicle damage assessment and annotation document, thereby improving the generation efficiency and accuracy of the damage assessment and annotation document of the target vehicle.
[0124] Optionally, the image recognition model includes: a multi-task neural network model obtained based on multi-task learning; the image recognition module 608 is specifically used to:
[0125] Extracting key video frames from the damaged area video frames in the vehicle damage video clip as video frame images to be identified;
[0126] The multi-task neural network model is used to perform component damage detection and image clarity detection on the video frame image to be identified, so as to obtain component damage detection results and image clarity detection results of the target vehicle; wherein the component damage detection results include: component damage type and damage location information.
[0127] Optionally, the damage image recognition result includes: component damage detection result and image clarity detection result; the vehicle damage assessment module 610 is specifically used to:
[0128] For each of the vehicle damage video clips, based on the image clarity detection result and the component damage detection result, selecting a target damage video frame in the vehicle damage video clip;
[0129] Based on the component damage detection result and the damage speech recognition result corresponding to the target damage video frame, the target damage video frame is damaged and marked to obtain a marked target damage video frame; wherein the marked target damage video frame includes: at least one of the target damage video frame, a damaged component identifier, a component damage type, and damage location information;
[0130] Based on the marked target damage video frames corresponding to each of the vehicle damage video clips, damage assessment annotation document information of the target vehicle is generated.
[0131] Optionally, the vehicle damage assessment module 610 is further specifically configured to:
[0132] In descending order of image clarity, selecting the damaged video frames with the highest image clarity from the vehicle damage video clips as candidate damaged video frames;
[0133] If the component damage detection result corresponding to the selected candidate damage video frame meets the preset constraint condition, the candidate damage video frame is determined as the target damage video frame;
[0134] If the component damage detection result corresponding to the selected candidate damage video frame does not meet the preset constraint condition, the next candidate damage video frame is selected until the target damage video frame is determined.
[0135] Optionally, the vehicle damage assessment module 610 is further specifically configured to:
[0136] Using a pre-trained image super-resolution model, super-resolution processing is performed on the target damaged video frame to obtain a target damaged video frame after super-resolution processing;
[0137] Based on the component damage detection result and the damage speech recognition result, damage marking is performed on the target damaged video frame after the super-resolution processing to obtain a marked target damaged video frame.
[0138] Optionally, the video segment extraction module 606 is specifically configured to:
[0139] For each of the time mark information in the impaired speech recognition result, determine the start interception time information and the end interception time information according to the time mark information and the preset video interception window;
[0140] In the vehicle damage video information, the vehicle damage video segment corresponding to the time mark information is extracted with the start interception time information as the starting point and the end interception time information as the end point.
[0141] Optionally, the above device further includes a first model training module, which is used to:
[0142] Acquire a speech training sample set; wherein the speech training sample set includes: a public speech training sample set and a vehicle detection speech training sample set;
[0143] The deep learning method is used to train the initial neural network model based on the speech training sample set to obtain a trained speech recognition model.
[0144] Optionally, the above device further includes a second model training module, which is used to:
[0145] Acquire an image training sample set; wherein the image training sample set includes a plurality of image training samples, each of which includes: a correspondence between sample image information, an actual component damage label, and an actual image clarity label;
[0146] Based on the image training sample set, model training is performed on the initial multi-task neural network model obtained based on multi-task learning to obtain a trained multi-task neural network model;
[0147] The multi-task neural network model is determined as a pre-trained image recognition model.
[0148] Optionally, the multi-task neural network model includes a component damage detection task branch and an image clarity detection task branch, the component damage detection task branch corresponds to a first loss function, and the image clarity detection task branch corresponds to a second loss function;
[0149] The second model training module is specifically used for:
[0150] Based on the image training sample set and the multi-task neural network model, obtaining a predicted component damage label and a predicted image clarity label for each of the sample image information;
[0151] Based on the predicted component damage label and the actual component damage label, updating the first loss function; and based on the predicted image definition label and the actual image definition label, updating the second loss function;
[0152] Based on the updated first loss function and the second loss function, updating the total loss function of the multi-task neural network model;
[0153] Iteratively updating the model parameters of the multi-task neural network model by using a back propagation algorithm and based on the updated total loss function until the total loss function converges;
[0154] The multi-task neural network model obtained based on the model parameters corresponding to the total loss function when it converges is determined as the trained multi-task neural network model.
[0155] Optionally, the second model training module is further specifically used for:
[0156] Based on a preset weight dynamic adjustment formula, determine a first weight corresponding to the first loss function and a second weight corresponding to the second loss function; wherein the first weight is used to characterize the training difficulty of the component damage detection task branch, and the second weight is used to characterize the training difficulty of the image clarity detection task branch;
[0157] Based on the first loss function and its corresponding first weight and the second loss function and its corresponding second weight, updating the total loss function of the multi-task neural network model;
[0158] The preset weight dynamic adjustment formula is W = (1-K) γ logK, γ represents the influencing factor parameter, and K represents the prediction accuracy of the task branch.
[0159] The vehicle damage assessment device in one or more embodiments of the present specification, after acquiring the vehicle damage video information of the target vehicle, first performs voice recognition on the vehicle damage video information, extracts the vehicle damage video segment from the vehicle damage video information, and then performs image recognition on the vehicle damage video segment to obtain a damage image recognition result, so as to extract the required vehicle damage video frame from the damage video segment based on the damage image recognition result, and finally performs damage assessment annotation on the vehicle damage video frame based on the damage voice recognition result and the damage image recognition result, thereby realizing automatic completion of damage assessment annotation of the vehicle damage information based on the vehicle damage video information by combining voice recognition and image recognition, without the need for the user to manually annotate the vehicle damage image and create a vehicle damage assessment annotation document, thereby improving the generation efficiency and accuracy of the damage assessment annotation document of the target vehicle.
[0160] It should be noted that the embodiments of the vehicle damage assessment device in this specification and the embodiments of the vehicle damage assessment method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the corresponding vehicle damage assessment method mentioned above, and the repeated parts will not be repeated.
[0161] Furthermore, corresponding to the above Figures 2 to 5 Based on the same technical concept, one or more embodiments of this specification also provide a vehicle damage assessment device, which is used to perform the above-mentioned vehicle damage assessment method, such as Figure 7 shown.
[0162] The vehicle damage assessment device may have relatively large differences due to different configurations or performances, and may include one or more processors 701 and memory 702, and the memory 702 may store one or more storage applications or data. Among them, the memory 702 can be short-term storage or persistent storage. The application stored in the memory 702 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the vehicle damage assessment device. Furthermore, the processor 701 can be configured to communicate with the memory 702 to execute a series of computer executable instructions in the memory 702 on the vehicle damage assessment device. The vehicle damage assessment device may also include one or more power supplies 703, one or more wired or wireless network interfaces 704, one or more input and output interfaces 705, one or more keyboards 706, etc.
[0163] In a specific embodiment, the vehicle damage assessment device includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions for the vehicle damage assessment device, and the one or more programs are configured to be executed by one or more processors, including the following computer executable instructions:
[0164] Acquire vehicle damage video information of the target vehicle, wherein the vehicle damage video information includes: damage description voice information for damaged parts of the vehicle and video frames of damaged areas;
[0165] Using a pre-trained speech recognition model, speech recognition is performed on the damage description speech information to obtain a damage speech recognition result of the target vehicle; wherein the damage speech recognition result includes: time stamp information, damaged component identification and component damage type;
[0166] extracting at least one vehicle damage video segment from the vehicle damage video information based on the damage speech recognition result;
[0167] Using a pre-trained image recognition model, image recognition is performed on the damaged area video frame in the vehicle damage video clip to obtain a damaged image recognition result of the target vehicle;
[0168] Based on the damage speech recognition result and the damage image recognition result, damage assessment and annotation document information of the target vehicle is generated.
[0169] The vehicle damage assessment device in one or more embodiments of the present specification, after acquiring the vehicle damage video information of the target vehicle, first performs voice recognition on the vehicle damage video information, extracts the vehicle damage video segment from the vehicle damage video information, and then performs image recognition on the vehicle damage video segment to obtain a damage image recognition result, so as to extract the required vehicle damage video frame from the damage video segment based on the damage image recognition result, and finally performs damage assessment annotation on the vehicle damage video frame based on the damage voice recognition result and the damage image recognition result, thereby realizing the automatic completion of the damage assessment annotation of the vehicle damage information based on the vehicle damage video information by combining voice recognition and image recognition, without the need for the user to manually annotate the damage on the vehicle damage image and prepare a vehicle damage assessment annotation document, thereby improving the generation efficiency and accuracy of the damage assessment annotation document of the target vehicle.
[0170] It should be noted that the embodiments of the vehicle damage assessment equipment in this specification and the embodiments of the vehicle damage assessment method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the corresponding vehicle damage assessment method mentioned above, and the repeated parts will not be repeated.
[0171] Furthermore, corresponding to the above Figures 2 to 5 The method shown is based on the same technical concept. One or more embodiments of this specification also provide a storage medium for storing computer executable instructions. In a specific embodiment, the storage medium can be a USB flash drive, an optical disk, a hard disk, etc. When the computer executable instructions stored in the storage medium are executed by the processor, the following process can be implemented:
[0172] Acquire vehicle damage video information of the target vehicle, wherein the vehicle damage video information includes: damage description voice information for damaged parts of the vehicle and video frames of damaged areas;
[0173] Using a pre-trained speech recognition model, speech recognition is performed on the damage description speech information to obtain a damage speech recognition result of the target vehicle; wherein the damage speech recognition result includes: time stamp information, damaged component identification and component damage type;
[0174] extracting at least one vehicle damage video segment from the vehicle damage video information based on the damage speech recognition result;
[0175] Using a pre-trained image recognition model, image recognition is performed on the damaged area video frame in the vehicle damage video clip to obtain a damaged image recognition result of the target vehicle;
[0176] Based on the damage speech recognition result and the damage image recognition result, damage assessment and annotation document information of the target vehicle is generated.
[0177] When the computer executable instructions stored in the storage medium in one or more embodiments of the present specification are executed by the processor, after obtaining the vehicle damage video information of the target vehicle, voice recognition is first performed on the vehicle damage video information, and a vehicle damage video clip is extracted from the vehicle damage video information. Then, image recognition is performed on the vehicle damage video clip to obtain a damage image recognition result, so as to extract the required vehicle damage video frame from the damage video clip based on the damage image recognition result, and finally, damage assessment and annotation are performed on the vehicle damage video frame based on the damage voice recognition result and the damage image recognition result, thereby realizing automatic completion of damage assessment and annotation of vehicle damage information based on the vehicle damage video information by combining voice recognition and image recognition, without the need for the user to manually annotate the vehicle damage image and create a vehicle damage assessment and annotation document, thereby improving the generation efficiency and accuracy of the damage assessment and annotation document of the target vehicle.
[0178] It should be noted that the embodiment of the storage medium in this specification and the embodiment of the vehicle damage assessment method in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the aforementioned corresponding vehicle damage assessment method, and the repeated parts will not be repeated.
[0179] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0180] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0181] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0182] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0183] For the convenience of description, the above devices are described in various units according to their functions. Of course, when implementing one or more of the present specification, the functions of each unit can be implemented in the same or more software and / or hardware.
[0184] It will be appreciated by those skilled in the art that one or more embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, one or more of this specification may be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more of this specification may be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0185] One or more of the present specification is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to one or more embodiments of the present specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0188] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0189] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.
[0190] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0191] It should also be noted that the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "includes one..." do not exclude the presence of other identical elements in the process, method, commodity or device including the elements. It should be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system or computer program product. Therefore, one or more of this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0192] One or more of the present specification may be described in the general context of computer executable instructions executed by a computer, such as a program module. Generally, a program module includes routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more of the present specification may also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices. The various embodiments in the present specification are described in a progressive manner, and the same and similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above description is only one or more embodiments of the present specification, and is not intended to limit one or more of the present specification. For those skilled in the art, one or more of the present specification may have various changes and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of one or more of the present specification should be included in the scope of one or more claims of the present specification.
Claims
1. A vehicle damage assessment method, characterized in that: include: Acquire vehicle damage video information of the target vehicle, wherein the vehicle damage video information includes: damage description voice information for damaged parts of the vehicle and video frames of damaged areas; Using a pre-trained speech recognition model, speech recognition is performed on the damage description speech information to obtain a damage speech recognition result of the target vehicle; wherein the damage speech recognition result includes: time stamp information, damaged component identification and component damage type; extracting at least one vehicle damage video segment from the vehicle damage video information based on the damage speech recognition result; Using a pre-trained image recognition model, image recognition is performed on the damaged area video frame in the vehicle damage video clip to obtain a damaged image recognition result of the target vehicle; Generate damage assessment and annotation document information of the target vehicle based on the damage speech recognition result and the damage image recognition result; The pre-trained image recognition model is trained in the following way: Acquire an image training sample set; wherein the image training sample set includes a plurality of image training samples, each of which includes: a correspondence between sample image information, an actual component damage label, and an actual image clarity label; Based on the image training sample set, model training is performed on the initial multi-task neural network model obtained based on multi-task learning to obtain a trained multi-task neural network model; Determining the multi-task neural network model as a pre-trained image recognition model; The multi-task neural network model includes a component damage detection task branch and an image clarity detection task branch, wherein the component damage detection task branch corresponds to a first loss function, and the image clarity detection task branch corresponds to a second loss function; The method of performing model training on the initial multi-task neural network model obtained based on multi-task learning based on the image training sample set to obtain a trained multi-task neural network model includes: Based on the image training sample set and the multi-task neural network model, obtaining a predicted component damage label and a predicted image clarity label for each of the sample image information; Based on the predicted component damage label and the actual component damage label, updating the first loss function; and based on the predicted image definition label and the actual image definition label, updating the second loss function; Based on the updated first loss function and the second loss function, updating the total loss function of the multi-task neural network model; Iteratively updating the model parameters of the multi-task neural network model by using a back propagation algorithm and based on the updated total loss function until the total loss function converges; The multi-task neural network model obtained based on the model parameters corresponding to the total loss function when it converges is determined as the trained multi-task neural network model.
2. The method according to claim 1, characterized in that: The image recognition model includes: a multi-task neural network model obtained based on multi-task learning; The method of using a pre-trained image recognition model to perform image recognition on the damaged area video frame in the vehicle damage video clip to obtain a damaged image recognition result of the target vehicle includes: Extracting key video frames from the damaged area video frames in the vehicle damage video clip as video frame images to be identified; The multi-task neural network model is used to perform component damage detection and image clarity detection on the video frame image to be identified, so as to obtain component damage detection results and image clarity detection results of the target vehicle; wherein the component damage detection results include: component damage type and damage location information.
3. The method according to claim 1, characterized in that: The damage image recognition result includes: component damage detection result and image clarity detection result; The generating of damage assessment and annotation document information of the target vehicle based on the damage speech recognition result and the damage image recognition result includes: For each of the vehicle damage video clips, based on the image clarity detection result and the component damage detection result, selecting a target damage video frame in the vehicle damage video clip; Based on the component damage detection result and the damage speech recognition result corresponding to the target damage video frame, the target damage video frame is damaged and marked to obtain a marked target damage video frame; wherein the marked target damage video frame includes: at least one of the target damage video frame, a damaged component identifier, a component damage type, and damage location information; Based on the marked target damage video frames corresponding to each of the vehicle damage video clips, damage assessment annotation document information of the target vehicle is generated.
4. The method according to claim 3, characterized in that The selecting a target damaged video frame from the vehicle damaged video clip based on the image clarity detection result and the component damage detection result includes: In descending order of image clarity, selecting the damaged video frames with the highest image clarity from the vehicle damage video clips as candidate damaged video frames; If the component damage detection result corresponding to the selected candidate damage video frame meets the preset constraint condition, the candidate damage video frame is determined as the target damage video frame; If the component damage detection result corresponding to the selected candidate damage video frame does not meet the preset constraint condition, the next candidate damage video frame is selected until the target damage video frame is determined.
5. The method according to claim 3, characterized in that: The step of marking the target damaged video frame for damage based on the component damage detection result and the damaged speech recognition result corresponding to the target damaged video frame to obtain the marked target damaged video frame includes: Using a pre-trained image super-resolution model, super-resolution processing is performed on the target damaged video frame to obtain a target damaged video frame after super-resolution processing; Based on the component damage detection result and the damage speech recognition result, damage marking is performed on the target damaged video frame after the super-resolution processing to obtain a marked target damaged video frame.
6. The method according to claim 1, characterized in that The extracting at least one vehicle damage video clip from the vehicle damage video information based on the damage speech recognition result includes: For each of the time mark information in the impaired speech recognition result, determine the start interception time information and the end interception time information according to the time mark information and the preset video interception window; In the vehicle damage video information, the vehicle damage video segment corresponding to the time mark information is extracted with the start interception time information as the starting point and the end interception time information as the end point.
7. The method according to claim 1, characterized in that The pre-trained speech recognition model is trained in the following way: Acquire a speech training sample set; wherein the speech training sample set includes: a public speech training sample set and a vehicle detection speech training sample set; The deep learning method is used to train the initial neural network model based on the speech training sample set to obtain a trained speech recognition model.
8. The method according to claim 1, characterized in that: The updating of the total loss function of the multi-task neural network model based on the updated first loss function and the second loss function comprises: Based on a preset weight dynamic adjustment formula, determine a first weight corresponding to the first loss function and a second weight corresponding to the second loss function; wherein the first weight is used to characterize the training difficulty of the component damage detection task branch, and the second weight is used to characterize the training difficulty of the image clarity detection task branch; Based on the first loss function and its corresponding first weight and the second loss function and its corresponding second weight, updating the total loss function of the multi-task neural network model; The preset weight dynamic adjustment formula is W = (1-K) γ logK, γ represents the influencing factor parameter, and K represents the prediction accuracy of the task branch.
9. A vehicle damage assessment device, characterized in that: include: A video information acquisition module, used to acquire vehicle damage video information of a target vehicle, wherein the vehicle damage video information includes: damage description voice information for damaged vehicle parts and a video frame of a damaged area; A speech recognition module, used to perform speech recognition on the damage description speech information using a pre-trained speech recognition model to obtain a damage speech recognition result of the target vehicle; wherein the damage speech recognition result includes: time stamp information, damaged component identification and component damage type; A video segment extraction module, configured to extract at least one vehicle damage video segment from the vehicle damage video information based on the damage speech recognition result; An image recognition module is used to perform image recognition on the damaged area video frame in the vehicle damage video clip using a pre-trained image recognition model to obtain a damage image recognition result of the target vehicle; A vehicle damage assessment module, used to generate damage assessment annotation document information of the target vehicle based on the damage voice recognition result and the damage image recognition result; The device further comprises a second model training module, which is used to: Acquire an image training sample set; wherein the image training sample set includes a plurality of image training samples, each of which includes: a correspondence between sample image information, an actual component damage label, and an actual image clarity label; Based on the image training sample set, model training is performed on the initial multi-task neural network model obtained based on multi-task learning to obtain a trained multi-task neural network model; Determining the multi-task neural network model as a pre-trained image recognition model; The multi-task neural network model includes a component damage detection task branch and an image clarity detection task branch, wherein the component damage detection task branch corresponds to a first loss function, and the image clarity detection task branch corresponds to a second loss function; The second model training module is specifically used for: Based on the image training sample set and the multi-task neural network model, obtaining a predicted component damage label and a predicted image clarity label for each of the sample image information; Based on the predicted component damage label and the actual component damage label, updating the first loss function; and based on the predicted image definition label and the actual image definition label, updating the second loss function; Based on the updated first loss function and the second loss function, updating the total loss function of the multi-task neural network model; Iteratively updating the model parameters of the multi-task neural network model by using a back propagation algorithm and based on the updated total loss function until the total loss function converges; The multi-task neural network model obtained based on the model parameters corresponding to the total loss function when it converges is determined as the trained multi-task neural network model.
10. A vehicle damage assessment device, characterized in that: include: a processor, and a memory arranged to store computer executable instructions; The computer executable instructions, when executed, cause the processor to implement the steps of the method of any one of claims 1 to 8 above.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Vehicle rapid damage determination method based on simulation of vehicle damages
CN105550756A
Real-time vehicle damage and component detection method based on mobile phone video stream
CN108875648A
Image recognition and maintenance scheme estimated loss assessment system based on artificial intelligence
CN108876640A
System and method for identifying compromised vehicle in online-to-offline service
CN111369012A
