Voice scene recognition device and method, storage medium and electronic equipment

Abstract

The invention discloses a voice scene recognition device and method, a storage medium and electronic equipment, and belongs to the technical field of audio. The device is at least used for processing user voice information by using a preprocessing module in response to input of a user voice instruction; and performing secondary processing on the processed user voice information by using a large model to generate semi-structured data and / or structured data, and using a scene database module to retrieve a scene database based on the semi-structured data and / or the structured data to generate scene data; and at least after the semi-structured data and / or the structured data meet a preset condition, using a large model to generate structured scene data based on the scene data and the semi-structured data and / or the structured data. According to the method and the device, the long-tail scene data can be generated on the basis of the scene database and the user voice information at least when the long-tail scene data is collected, so that the comprehensiveness, the real-time performance and the accuracy of long-tail scene data collection are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of audio technology, and in particular to a speech scene recognition device, method, storage medium, and electronic device. Background Art

[0002] Currently, autonomous driving requires the collection of a huge amount of data (such as vehicle status, road conditions, driving habits, etc.), and autonomous driving needs to operate in various complex scenarios (such as urban roads, highways, rural roads, etc.). The data collection rules required for autonomous driving in different scenarios are different, which increases the difficulty of data collection. At the same time, autonomous driving also needs to obtain a large amount of long-tail scene data to improve the generalization ability of autonomous driving algorithms, but a large number of long-tail scenes are out of reach for single-vehicle autonomous driving. How to quickly capture the long-tail scene and record the relevant scene data when the vehicle encounters the long-tail scene is the current challenge of data collection.

[0003] At present, there are two common triggering methods for scene data collection: one is automatic triggering by vehicle perception, and the other is manual triggering (such as touch screen, keyboard input, etc.). However, the automatic triggering method of vehicle perception lacks comprehensive scene description content, and the collected content is not comprehensive enough; the manual triggering method has problems such as untimely triggering and low triggering efficiency. Summary of the invention

[0004] The embodiments of the present invention provide a voice scene recognition device, method, storage medium and electronic device to at least solve the problems of lack of comprehensive coverage of scene description content and incomplete collection in the existing long-tail scene data collection process of autonomous driving, and the problem of untimely triggering and low triggering efficiency in the manual triggering method, which is conducive to improving the comprehensiveness, real-timeness and result accuracy of the long-tail scene data collection process.

[0005] In a first aspect, an embodiment of the present invention provides a speech scene recognition device, wherein the speech scene recognition device is at least used for:

[0006] In response to the input of user voice commands, a preprocessing module is used to process the user voice information; and, based on the user voice information processed by the preprocessing module, a large model is used to perform secondary processing on the processed user voice information to generate semi-structured data and / or structured data; and, a scene database module is used to retrieve a scene database based on the semi-structured data and / or the structured data to generate scene data; and, at least after the semi-structured data and / or the structured data meet preset conditions, the large model is used to generate structured scene data based on the scene data, and the semi-structured data and / or the structured data.

[0007] Optionally, the voice scene recognition device is further used to, after the voice scene recognition device is turned on, change the collection state of the user voice information by the voice scene recognition device in response to each acquisition of the user voice instruction.

[0008] Optionally, the speech scene recognition device is further used to input the structured scene data into the scene database.

[0009] Optionally, the preprocessing module includes at least a text conversion module;

[0010] The text conversion module is at least used to process the user voice information to generate text information based on the user voice information.

[0011] Optionally, the large model includes at least a data processing module, an attention module and a fusion module;

[0012] The data processing module is at least used to generate the semi-structured data and / or the structured data based on the text information;

[0013] The attention module is at least used to evaluate the semi-structured data and / or the structured data and generate an evaluation result; and when the evaluation result does not meet the preset condition, re-input the corresponding semi-structured data and / or the structured data as input to the data processing module;

[0014] The fusion module is at least used for fusing the semi-structured data and / or the structured data and the corresponding scene data to generate the structured scene data when the evaluation result meets the preset condition.

[0015] Optionally, the scene database module includes at least a vectorization module and a scene information generation module;

[0016] The vectorization module is at least used to generate vectorized data based on the semi-structured data and / or the structured data;

[0017] The scene information generation module is at least used to retrieve the scene database to obtain relevant scene vector data based on the vectorized data; and to generate the scene data based on the relevant scene vector data; and, at least after the scene data is generated, send the scene data to the large model.

[0018] Optionally, the speech scene recognition device comprises at least an evaluation module and a structured processing module;

[0019] The evaluation module is at least used to screen each of the structured scene data to obtain structured scene data that meets a predetermined condition;

[0020] The structured processing module is at least used to perform structured correction on the structured scene data that meets the predetermined conditions.

[0021] In a second aspect, this embodiment further provides a method for speech scene recognition, which is performed by the speech scene recognition device described in the first aspect, and the method at least includes:

[0022] S1, using the preprocessing module to process user voice information in response to input of a user voice command by the voice scene recognition device;

[0023] S2, using the speech scene recognition device to perform secondary processing on the user speech information processed by the preprocessing module using a large model to generate semi-structured data and / or structured data;

[0024] S3, using the scene database module to search a scene database based on the semi-structured data and / or the structured data by the speech scene recognition device to generate scene data;

[0025] S4. The speech scene recognition device generates structured scene data using the large model based on the scene data, the semi-structured data and / or the structured data at least after the semi-structured data and / or the structured data meet a preset condition.

[0026] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium, which, when executed by a processor, implements the voice scene recognition method described in the second aspect.

[0027] In a fourth aspect, an embodiment of the present invention further provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the speech scene recognition method described in the second aspect.

[0028] In summary, the embodiments of the present invention provide a speech scene recognition device, method, storage medium and electronic device. First, the speech scene recognition device responds to the input of the user voice command and uses the preprocessing module to process the user voice information; secondly, the speech scene recognition device uses the large model to perform secondary processing on the processed user voice information based on the user voice information processed by the preprocessing module to generate semi-structured data and / or structured data; thirdly, the speech scene recognition device uses the scene database module to retrieve the scene database based on the semi-structured data and / or structured data to generate scene data; finally, the speech scene recognition device uses the large model to generate structured scene data based on the scene data, and the semi-structured data and / or structured data at least after the semi-structured data and / or structured data meet the preset conditions.

[0029] In the scene data collection process of the above-mentioned device, the voice scene recognition device can be embedded in the vehicle intelligent cockpit (the voice scene recognition device can include, for example, an application in the vehicle intelligent cockpit), and the user can be a driver or a passenger. Therefore, based on the technical solution provided by the embodiment of the present invention, when the driver needs to collect long-tail scene data, he can use voice commands to quickly wake up the application in the vehicle intelligent cockpit to collect user voice information, and then generate long-tail scene data based on the scene database and user voice information, at least solving the problem of the lack of comprehensive coverage of scene description content and incomplete collection in the existing autonomous driving long-tail scene data collection process of the vehicle perception automatic triggering method, and the problem of untimely triggering and low triggering efficiency in the manual triggering method, which is conducive to improving the comprehensiveness, real-timeness and result accuracy of the long-tail scene data collection process. It can be understood that data collection through voice can also reduce the difficulty of data collection and is easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 is a structural schematic diagram of a speech scene recognition device provided by an embodiment of the present invention;

[0032] Figure 2 is a schematic diagram of the architecture of a speech scene recognition device provided by an embodiment of the present invention;

[0033] Figure 3 It is an implementation flow chart of a speech scene recognition device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0035] Figure 1 is a structural diagram of a speech scene recognition device provided by an embodiment of the present invention, such as Figure 1 shown.

[0036] The speech scene recognition device 100 is at least used to, in response to the input of a user voice command, use the preprocessing module 130 to process the user voice information; and, based on the user voice information processed by the preprocessing module 130, use the large model 120 to perform secondary processing on the processed user voice information to generate semi-structured data and / or structured data; and, use the scene database module 110 to retrieve the scene database 110 based on the semi-structured data and / or structured data to generate scene data; and, at least after the semi-structured data and / or structured data meet preset conditions, use the large model 120 to generate structured scene data based on the scene data, and the semi-structured data and / or structured data.

[0037] The user voice command information may be a specific voice information that can be recognized or understood by the voice scene recognition device 100, so that the voice scene recognition device 100 completes the corresponding action after recognizing or understanding the voice information. Accordingly, the voice scene recognition device 100 may also start automatically when the vehicle starts.

[0038] The preprocessing module 130 is used to process the user voice information. After collecting the user voice information, the speech scene recognition device 100 converts the user voice information into a data type that can be processed by the large model 120 (for example, the user voice information can be converted into text information). The large model 120 can be a model architecture that can convert the user voice information (for example, it can be text information) processed by the speech scene recognition device 100 into semi-structured data and / or structured data (including all the features of the aforementioned text information); the large model 120 can be, for example, a Transformer model (a Transformer model is a deep learning model architecture for natural language and other sequence-to-sequence task processing). The scene database module 110 is used to retrieve the scene database 110 based on the semi-structured data and / or structured data. Generating scene data can mean that the speech scene recognition device 100 retrieves the scene data in the scene database 110 that matches the features of the aforementioned semi-structured data and / or structured data according to the semi-structured data and / or structured data generated by the Transformer model.

[0039] In addition, the preset condition may refer to a user-defined or manufacturer-preset scoring standard. Semi-structured data and / or structured data satisfying the preset condition may mean that, assuming that the user manually sets the scoring standard to 0.8, if the score of a certain group of semi-structured data and / or structured data by the large model 120 is higher than 0.8 points, then the group of semi-structured data and / or structured data is considered to be excellent (i.e., it satisfies the preset condition). On the contrary, if the score of the semi-structured data and / or structured data by the large model 120 is lower than 0.8 points, then the group of semi-structured data and / or structured data is considered to be poor (i.e., it does not meet the preset condition).

[0040] Using the big model 120 to generate structured scene data based on scene data, as well as semi-structured data and / or structured data may mean that the speech scene recognition device 100 uses the big model 120 to generate structured scene data based on semi-structured data and / or structured data that meet preset conditions and scene data generated by retrieving the scene database 110 based on the aforementioned semi-structured data and / or structured data, and fuses them together to generate structured scene data.

[0041] In a specific implementation, first, the driver presses the car start button to start the car. Further, the voice scene recognition device 100 (hereinafter referred to as the recognition device) automatically starts with the start of the car. Further, during the driving of the car, it passes through a section involving a long-tail scene. At this time, it is raining heavily outside the car, and the driver wants to collect the current long-tail scene data. Further, the driver issues a user voice command of "scene collection starts". Further, after receiving the user voice command of "scene collection starts", the recognition device starts to collect user voice information. Further, the user voice information collected by the recognition device is "a new scene appears. Now, it is raining heavily. The vehicle turns left and an oncoming vehicle passes quickly. I stepped on the brakes and the vehicle skidded. It is a simple intersection with two lanes in both directions. Please record this scene and the scene collection ends". Further, when the recognition device receives the user voice command of "scene collection ends", it stops collecting user voice information and generates text information of the user voice information. Furthermore, the recognition device uses the Transformer model to generate semi-structured data and / or structured data of "heavy rain, vehicle turning left, oncoming vehicle passing quickly, braking, vehicle skidding, and oncoming two-lane intersection" based on text information, and sets the score of this group of data to 0.81 (the Transformer model will output multiple groups of semi-structured data and / or structured data and the scores corresponding to the groups of data. This embodiment only takes the output of one group as an example for explanation). Furthermore, the recognition device retrieves the scene database based on the aforementioned semi-structured data and / or structured data. Further, the recognition device obtains the scene data of "heavy rain, vehicle turning left, braking, vehicle skidding, oncoming two-lane intersection, vehicle speed of 10km / h, and traffic light status of green". Furthermore, after the recognition device determines that the score of the aforementioned semi-structured data and / or structured data is greater than the preset requirement of 0.8 points, the recognition device generates structured scene data of "heavy rain, vehicle turning left, oncoming vehicle passing quickly, braking, vehicle side sliding, oncoming two-lane intersection, vehicle speed 10km / h, traffic light status green" by using the Transformer model based on the aforementioned semi-structured data and / or structured data and scene data fusion. At this point, the collection of long-tail scene data by this round of recognition device is completed.

[0042] In summary, based on the technical solution provided by this embodiment, when the driver needs to collect long-tail scene data, he can use voice commands to quickly wake up the application in the smart cockpit of the vehicle to collect user voice information, and then generate long-tail scene data based on the scene database and user voice information, at least solving the existing problem of long-tail scene data collection in autonomous driving, that the vehicle perception automatic triggering method lacks comprehensive coverage of scene description content and incomplete collection, and the manual triggering method is not timely and has low triggering efficiency, which is conducive to improving the comprehensiveness, real-timeness and result accuracy of the long-tail scene data collection process. It can be understood that data collection through voice can also reduce the difficulty of data collection and is easy to implement.

[0043] It should be noted that, in addition to long-tail scenarios, the application scenarios of the voice scene recognition device can also be other scenarios where scene data collection is required, such as ship driving scenarios, etc. In other words, the voice scene recognition device provided in this embodiment can also be applied to other technical fields to meet the collection requirements of corresponding equipment, which will not be repeated.

[0044] Based on the above embodiments or implementations, this embodiment adds to and refines the above-mentioned speech scene recognition device 100, but does not limit the present invention. Figure 2 is a schematic diagram of the architecture of the speech scene recognition device provided by an embodiment of the present invention, see Figure 2 In a specific implementation, optionally, the speech scene recognition device 100 is also used to change the collection state of the user voice information by the speech scene recognition device 100 in response to each acquisition of the user voice command after the speech scene recognition device 100 is turned on.

[0045] Among them, in response to each acquisition of a user voice command, changing the collection status of the user voice information by the voice scene recognition device 100 may mean that, after the voice scene recognition device 100 acquires a voice command representing the start of collecting user voice information, the voice scene recognition device 100 switches from a standby state to a working state (or from background operation to foreground operation) and starts collecting user voice information, and after the voice scene recognition device 100 acquires a user voice command representing the stop of collecting user voice information, the voice scene recognition device 100 stops collecting user voice information.

[0046] Exemplarily, the speech scene recognition device 100 is in a standby state after being started; at a certain moment, the user issues a user voice command of "start scene collection", and after the speech scene recognition device 100 obtains the user voice command of "start scene collection", it switches from the standby state to the working state and starts collecting user voice information; at another moment, the speech scene recognition device 100 stops collecting user voice information after obtaining the user voice command of "end scene collection".

[0047] In another specific implementation, optionally, the speech scene recognition device 100 is further configured to input the structured scene data into a scene database.

[0048] In yet another specific implementation, optionally, the preprocessing module 130 includes at least a text conversion module 210;

[0049] The text conversion module 210 is at least used to process the user voice information to generate text information based on the user voice information.

[0050] Among them, the specific process of the text conversion module 210 can be that when the user voice information collected by the voice scene recognition device 100 is "Now, it is raining heavily, the vehicle is turning left and there is an oncoming vehicle driving through quickly, it is a simple intersection with two lanes in both directions", the text conversion module 210 converts the collected user voice information into text information to obtain the text information "Now, it is raining heavily, the vehicle is turning left and there is an oncoming vehicle driving through quickly, it is a simple intersection with two lanes in both directions".

[0051] It is understandable that the processing flow of the aforementioned text conversion module 210 may be processed by a natural language processing (Natural Language Processing, NLP) model.

[0052] In yet another specific embodiment, optionally, the large model 120 includes at least a data processing module 220 , an attention module 230 , and a fusion module 240 ;

[0053] The data processing module 220 is at least used to generate semi-structured data and / or structured data based on the text information;

[0054] The attention module 230 is at least used to evaluate the semi-structured data and / or structured data and generate an evaluation result; and when the evaluation result does not meet the preset conditions, re-input the corresponding semi-structured data and / or structured data as input to the data processing module;

[0055] The fusion module 240 is at least used to fuse the semi-structured data and / or structured data and their corresponding scene data to generate structured scene data when the evaluation result meets a preset condition.

[0056] Among them, the attention module 230 evaluates the semi-structured data and / or structured data and generates an evaluation result, which may mean that the attention module 230 performs a comprehensive evaluation based on multiple groups of semi-structured and / or structured data and their corresponding confidence information output by the data processing module 220, and the correlation between the scene data corresponding to each group of semi-structured and / or structured data (the correlation can be calculated using the Jaccard algorithm).

[0057] Exemplarily, the attention module 230 may evaluate the semi-structured data and / or structured data by multiplying the confidence of each group of semi-structured and / or structured data by the correlation between each group of semi-structured and / or structured data and the corresponding scene data, and then weighted averaging each product to obtain a weighted average, which is the evaluation result. The weight may be user-defined or reduced in sequence in the order of data output (for example, when the data processing module 220 generates 4 groups of data, the weights corresponding to each group of data are 0.4, 0.3, 0.2, and 0.1, respectively).

[0058] It should be noted that the aforementioned Jaccard algorithm is translated as Jaccard algorithm, which is a statistical method used to measure the similarity between two sets.

[0059] It can be understood that re-inputting the corresponding semi-structured data and / or structured data as input into the data processing module 220 may mean re-inputting the semi-structured data and / or structured data generated in this round into the data processing module 220 using the feedback processing interface of the data processing module 220 (here is a positive feedback process), and regenerating the semi-structured data and / or structured data.

[0060] Fusion to generate structured scene data may refer to fusing semi-structured and / or structured data and corresponding scene data using a fusion algorithm to generate structured scene data. The fusion algorithm may be, for example, a cross fusion algorithm.

[0061] In yet another specific implementation, optionally, the scene database module 110 includes at least a vectorization module 240 and a scene information generation module 250;

[0062] The vectorization module 240 is at least used to generate vectorized data based on the semi-structured data and / or the structured data;

[0063] The scene information generation module 250 is at least used to retrieve the scene database 110 to obtain relevant scene vector data based on the vectorized data; and to generate scene data based on the relevant scene vector data; and, at least after the scene data is generated, send the scene data to the large model 120.

[0064] Among them, generating vectorized data based on semi-structured data and / or structured data may refer to converting the data format or data type of the semi-structured data and / or structured data into a data format or data type that can be used by the scene database 110 (the data after conversion contains all the features of the aforementioned semi-structured data and / or structured data). Exemplarily, generating vectorized data based on semi-structured data and / or structured data may refer to vectorizing the semi-structured data of "oncoming vehicles passing quickly" to generate a multi-dimensional vector (the multi-dimensional vector represents "oncoming vehicles passing quickly" in the scene data database 110).

[0065] In another specific implementation, optionally, the speech scene recognition device 100 includes at least an evaluation module 270 and a structured processing module 280;

[0066] The evaluation module 270 is at least used to screen each structured scene data to obtain structured scene data that meets a predetermined condition;

[0067] The structured processing module 280 is at least used to perform structural correction on the structured scene data that meets predetermined conditions.

[0068] The evaluation module 270 screens each structured scene data to obtain structured scene data that meets the predetermined conditions, which may refer to obtaining the confidence of the semi-structured data and / or structured data corresponding to the structured scene data when generating the structured scene data, and determining whether the confidence meets the predetermined conditions. The structural correction of the structured scene data that meets the predetermined conditions may refer to removing and correcting data information that is irrelevant to the scene information (for example, it may be the clothing information of passers-by) in the structured scene data.

[0069] For example, the actual working principle of the voice scene recognition device provided in this embodiment can be specifically described as follows:

[0070] First, after the driver presses the car start button to start the car, the voice scene recognition device 100 (hereinafter referred to as the recognition device) automatically starts with the start of the car and is placed in the background. Further, the driver encounters a long-tail scene while the car is driving. At this time, the driver wants to collect the current long-tail scene information. Further, the driver issues a user voice command of "scene collection starts". Further, after obtaining the user voice command of "scene collection starts", the recognition device switches from the background operation state to the foreground operation state. Further, the recognition device collects the driver's user voice information as "a new scene appears, a two-way lane, the left road is under construction, I drive the vehicle on the rightmost road and wait for the straight-going traffic light at the intersection ahead to turn green and then drive straight away from the intersection. At this time, there are 48 seconds before the green light. I hear the sound of an ambulance alarm from behind. I need to avoid the ambulance. I step on the accelerator. After the car drives 5m to the right front, I step on the brake, the vehicle stops running, and then the ambulance passes normally. Please record this scene, and the scene collection is completed." Furthermore, after the recognition device obtains the user voice command of "scene collection ends", it stops collecting the user voice information. Furthermore, the recognition device converts the collected user voice information into text information using the NLP model (this step corresponds to Figure 2 Further, the recognition device inputs the text information into the Transformer model (corresponding to Figure 2 After the large model 120 shown in the figure, the Transformer model generates multiple sets of semi-structured data and / or structured data (this step specifically corresponds to Figure 2 The data processing module 220 in the example above includes: "two-way lane, left road construction, vehicle on the rightmost road, waiting for the traffic light to turn green at the intersection ahead, driving straight away from the intersection, 48 seconds before the green light, ambulance alarm sounds, avoid the ambulance, step on the accelerator, start the car and make the car drive 5 meters to the right front, step on the brake, stop the vehicle and allow the ambulance to pass normally", "two-way lane, left road under construction, driving the vehicle on the rightmost road, red and green at the intersection, driving straight away from the intersection, 48 seconds before the green light, alarm sounds, avoid, start the car and make the car drive 5 meters to the right front and stop the vehicle", ..., and corresponding confidence levels of 0.85, 0.71, ... (the Transformer model will generate multiple sets of semi-structured data and / or structured data and their corresponding confidence values ​​based on the above text information, which are not listed here in full). Further, the Transformer model is connected to the scene database module 110 according to its own retrieval interface, and the scene database 290 in the scene database module 110 is used as a knowledge base. Furthermore, the scene database module 110 performs vectorization processing on multiple sets of semi-structured data and / or structured data to generate multiple sets of vectorized data (this step corresponds to Figure 2The vectorization module 240 in (each set of vectorized data contains all the features of the corresponding semi-structured data and / or structured data). Further, the scene database module 110 retrieves the scene database 290 according to the first set of vectorized data, obtains the relevant scene vector data and generates scene data (this step corresponds to Figure 2 The scene information generation module 250 in it): "two-way lane, driving the vehicle on the rightmost road, driving straight away from the intersection, waiting for the red light to turn green, the fire truck alarm sound, avoiding the fire truck, starting the car to the right front, avoiding pedestrians, stopping the vehicle and the fire truck passing normally", and reminding the driver that the scene data was recorded by the user named "2794" on July 25, 2023; and, the scene database module 110 retrieves the scene data according to the second set of vectorized data: "Driving the vehicle on the rightmost road, driving straight away from the intersection, green light, avoiding pedestrians, stopping the vehicle, and the vehicle passing normally", and reminding the driver that the scene data was recorded by the user named "2178" on April 24, 2023; and,... Further, the scene database module 110 sends the scene data to the Transformer model. Further, the Transformer model evaluates multiple sets of semi-structured data and / or structured data, and their corresponding scene data (this step corresponds to Figure 2Attention module 230 in the attention module 230). Further, the Transformer model uses the Jaccard algorithm to calculate the correlation between the semi-structured data and / or structured data and the corresponding scene data, which are 0.81, 0.75, ... respectively. Further, the Transformer model multiplies the confidence of each group of data by the corresponding correlation, and then weights each product (the order of weights can depend on the order in which the semi-structured data and / or structured data are generated, and the size of the weights can be reduced in turn) to obtain a weighted average value of 0.83. Further, since the obtained evaluation value of 0.83 (i.e., the weighted average value) is greater than the preset value of 0.8 (if the evaluation value is lower than 0.8, the current round of semi-structured data and / or structured data is re-input into the Transformer model through the feedback interface of the Transformer model for processing). Furthermore, the Transformer model generates structured scene data using a cross-fusion algorithm based on semi-structured data and / or structured data and their corresponding scene data: "two-way lane, left road under construction, driving the vehicle on the rightmost road, waiting for the red light to turn green, driving straight away from the intersection, ambulance alarm sound, avoid the ambulance, step on the accelerator, start the car and make the car drive 5 meters to the right front, avoid pedestrians, step on the brakes, stop the vehicle and the ambulance to pass normally", "two-way lane, left road under construction, driving the vehicle on the rightmost road, going straight red and green, 48 seconds until the green light, alarm sound, avoid, start the car and make the car drive 5 meters to the right front, stop the vehicle and drive normally", ... (this step corresponds to Figure 2 The fusion module 260 in the above embodiment retains the confidence level corresponding to the semi-structured data and / or structured data. Further, the recognition device selects the structured scene data with a confidence level exceeding 0.8 (this step corresponds to Figure 2 Evaluation module 270 in the example), namely, “two-way lane, road construction on the left, driving the vehicle on the rightmost road, waiting for the red light to change to green, driving straight away from the intersection, ambulance sirens sounding, avoiding the ambulance, stepping on the accelerator, starting the car and driving it 5m to the right front, avoiding pedestrians, stepping on the brakes, stopping the vehicle and allowing the ambulance to pass normally”. Further, the recognition device removes and corrects the aforementioned structured scene data, and finally generates “two-way lane, road construction on the left, vehicle on the rightmost road, waiting for the red light to change to green, driving straight away from the intersection, ambulance sirens sounding, avoiding pedestrians, avoiding the ambulance, stepping on the accelerator, starting the car and driving it to the right front, stepping on the brakes, stopping the vehicle, and allowing the ambulance to pass normally” (this step corresponds to Figure 2 Further, the recognition device stores the aforementioned structured scene data and the timestamp of the current moment in the scene database 290 of the scene database module 110. At this point, the collection of the long-tail scene data is completed, and the recognition device is put back into the background operation.

[0071] In summary, when collecting long-tail scene data, this embodiment can not only use voice to quickly wake up the voice scene collection device, but also introduce the existing autonomous driving scene library as a knowledge base in the process of generating long-tail scene data, at least solving the problem of the lack of comprehensive coverage of scene description content and incomplete collection in the vehicle perception automatic triggering method in the existing autonomous driving long-tail scene data collection process, as well as the problem of untimely triggering and low triggering efficiency in the manual triggering method, which is conducive to improving the comprehensiveness, real-timeness and result accuracy of the long-tail scene data collection process. It can be understood that data collection through voice can also reduce the difficulty of data collection and is easy to implement.

[0072] The embodiment of the present invention also provides a method for voice scene recognition. Figure 3 It is a flow chart of a voice scene recognition method provided by an embodiment of the present invention. This embodiment can be applicable to the long-tail scene data collection scenarios of various types of vehicles, such as fuel vehicles, pure electric vehicles, oil-electric hybrid vehicles, plug-in hybrid vehicles, extended-range electric vehicles, driverless vehicles, etc. The voice scene recognition method can be, but is not limited to, executed by the voice scene recognition device in the embodiment of the present invention as the execution subject, and the execution subject can be implemented in software and / or hardware. Figure 3 As shown, the voice scene recognition method includes at least the following steps:

[0073] S1. In response to input of a user voice command, a voice scene recognition device uses a preprocessing module to process user voice information.

[0074] S2. Based on the user voice information processed by the preprocessing module, the voice scene recognition device uses a large model to perform secondary processing on the processed user voice information to generate semi-structured data and / or structured data.

[0075] S3. Using a scene database module, the voice scene recognition device searches a scene database based on semi-structured data and / or structured data to generate scene data.

[0076] S4. After at least the semi-structured data and / or the structured data meet preset conditions, the speech scene recognition device uses a large model to generate structured scene data based on the scene data, and the semi-structured data and / or the structured data.

[0077] Optionally, the voice scene recognition method further includes at least:

[0078] After the voice scene recognition device is turned on, in response to each acquisition of a user voice command, the voice scene recognition device changes the state of collecting user voice information.

[0079] Optionally, the voice scene recognition method further includes at least:

[0080] The structured scene data is input into the scene database through the voice scene recognition device.

[0081] Optionally, the preprocessing module includes at least a text conversion module;

[0082] Step S1 at least includes:

[0083] In response to the input of the user voice instruction, the user voice information is processed by the text conversion module to generate text information based on the user voice information.

[0084] Optionally, the large model includes at least a data processing module, an attention module and a fusion module;

[0085] Step S2 at least includes:

[0086] Generate semi-structured data and / or structured data based on the text information through a data processing module;

[0087] Step S4 at least includes:

[0088] Evaluate the semi-structured data and / or structured data through the attention module and generate evaluation results;

[0089] When the evaluation result does not meet the preset conditions, the corresponding semi-structured data and / or structured data are re-input as input to the data processing module through the attention module;

[0090] When the evaluation result meets the preset conditions, the semi-structured data and / or structured data and their corresponding scene data are fused through the fusion module to generate structured scene data.

[0091] Optionally, the scene database module includes at least a vectorization module and a scene information generation module;

[0092] Step S3 at least includes:

[0093] Generate vectorized data based on the semi-structured data and / or the structured data through a vectorization module;

[0094] Retrieving the scene database to obtain relevant scene vector data based on the vectorized data through the scene information generation module;

[0095] Generate scene data based on relevant scene vector data through a scene information generation module;

[0096] After the scene data is generated, it is sent to the large model through the scene information generation module.

[0097] Optionally, the speech scene recognition device includes at least an evaluation module and a structured processing module;

[0098] The speech scene recognition method at least includes:

[0099] Screening each structured scene data through the evaluation module to obtain structured scene data that meets predetermined conditions;

[0100] The structured scene data that meets the predetermined conditions is structurally modified through the structured processing module.

[0101] The technical solution provided in this embodiment is as follows: first, the voice scene recognition device responds to the input of the user voice command and uses the preprocessing module to process the user voice information; secondly, the voice scene recognition device uses the large model to perform secondary processing on the user voice information processed by the preprocessing module to generate semi-structured data and / or structured data; then, the voice scene recognition device uses the scene database module to retrieve the scene database based on the semi-structured data and / or structured data to generate scene data; finally, the voice scene recognition device uses the large model to generate structured scene data based on the scene data, and the semi-structured data and / or structured data at least after the semi-structured data and / or structured data meet the preset conditions.

[0102] Based on the technical solution provided by this embodiment, when the driver needs to collect long-tail scene data, he can use voice commands to quickly wake up the application in the smart cockpit of the vehicle to collect user voice information, and then generate long-tail scene data based on the scene database and user voice information, at least solving the existing problem of long-tail scene data collection in autonomous driving, that the vehicle perception automatic triggering method lacks comprehensive coverage of scene description content and incomplete collection, and the manual triggering method is not timely and has low triggering efficiency, which is conducive to improving the comprehensiveness, real-timeness and result accuracy of the long-tail scene data collection process. It is understandable that data collection through voice can also reduce the difficulty of data collection and is easy to implement.

[0103] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program that can be executed by an electronic device. When the computer program runs on the electronic device, the electronic device executes the speech scene recognition method provided in the embodiment of the present application: a speech scene recognition device responds to the input of a user voice command and uses a preprocessing module to process the user voice information; the speech scene recognition device uses a large model to perform secondary processing on the processed user voice information based on the user voice information processed by the preprocessing module to generate semi-structured data and / or structured data; the speech scene recognition device uses a scene database module to retrieve a scene database based on the semi-structured data and / or structured data to generate scene data; the speech scene recognition device uses a large model to generate structured scene data based on the scene data, and the semi-structured data and / or structured data at least after the semi-structured data and / or structured data meet preset conditions.

[0104] An embodiment of the present invention also provides an electronic device, the electronic device comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the speech scene recognition method provided in the embodiment of the present application: a speech scene recognition device responds to the input of a user voice command and uses a preprocessing module to process the user voice information; the speech scene recognition device uses a large model to perform secondary processing on the processed user voice information based on the user voice information processed by the preprocessing module to generate semi-structured data and / or structured data; the speech scene recognition device uses a scene database module to retrieve a scene database based on the semi-structured data and / or structured data to generate scene data; the speech scene recognition device uses a large model to generate structured scene data based on the scene data, and the semi-structured data and / or structured data at least after the semi-structured data and / or structured data meet preset conditions.

[0105] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A speech scene recognition device, characterized in that: The speech scene recognition device is at least used for: In response to the input of the user voice command, the user voice information is processed using a preprocessing module; and based on the user voice information processed by the preprocessing module, the processed user voice information is secondary processed using a large model to generate semi-structured data and / or structured data; and, using a scenario database module to search a scenario database based on the semi-structured data and / or the structured data to generate scenario data; And, at least after the semi-structured data and / or the structured data meet a preset condition, the large model is used to generate structured scene data based on the scene data, and the semi-structured data and / or the structured data.

2. The speech scene recognition device according to claim 1, characterized in that: The voice scene recognition device is also used to change the collection state of the user voice information by the voice scene recognition device in response to each acquisition of the user voice instruction after the voice scene recognition device is turned on.

3. The speech scene recognition device according to claim 1, characterized in that: The speech scene recognition device is further used to input the structured scene data into the scene database.

4. The speech scene recognition device according to claim 1, characterized in that: The preprocessing module at least includes a text conversion module; The text conversion module is at least used to process the user voice information to generate text information based on the user voice information.

5. The speech scene recognition device according to claim 4, characterized in that: The large model at least includes a data processing module, an attention module and a fusion module; The data processing module is at least used to generate the semi-structured data and / or the structured data based on the text information; The attention module is at least used to evaluate the semi-structured data and / or the structured data and generate an evaluation result; and when the evaluation result does not meet the preset condition, re-input the corresponding semi-structured data and / or the structured data as input to the data processing module; The fusion module is at least used for fusing the semi-structured data and / or the structured data and the corresponding scene data to generate the structured scene data when the evaluation result meets the preset condition.

6. The speech scene recognition device according to claim 5, characterized in that: The scene database module at least includes a vectorization module and a scene information generation module; The vectorization module is at least used to generate vectorized data based on the semi-structured data and / or the structured data; The scene information generation module is at least used to retrieve the scene database to obtain relevant scene vector data based on the vectorized data; and to generate the scene data based on the relevant scene vector data; and, at least after the scene data is generated, send the scene data to the large model.

7. The speech scene recognition device according to claim 6, characterized in that: The speech scene recognition device at least includes an evaluation module and a structured processing module; The evaluation module is at least used to screen each of the structured scene data to obtain structured scene data that meets a predetermined condition; The structured processing module is at least used to perform structural correction on the structured scene data that meets the predetermined conditions.

8. A method for speech scene recognition, characterized in that: The method is performed by using the speech scene recognition device according to claim 1, and the method at least comprises: S1, using the preprocessing module to process user voice information in response to input of a user voice command by the voice scene recognition device; S2, based on the user voice information processed by the preprocessing module, the voice scene recognition device performs secondary processing on the processed user voice information using a large model to generate semi-structured data and / or structured data; S3, using the scene database module to search a scene database based on the semi-structured data and / or the structured data by the speech scene recognition device to generate scene data; S4. The speech scene recognition device generates structured scene data using the large model based on the scene data, the semi-structured data and / or the structured data at least after the semi-structured data and / or the structured data meet a preset condition.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the speech scene recognition method described in claim 8 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the speech scene recognition method described in claim 8.

Images