Voice interaction method and device based on generative large model

Abstract

The invention provides a voice interaction method and device based on a generative large model. The problem that an existing voice assistant cannot effectively recognize a voice instruction of a user through the generative large model is effectively solved. The method comprises the following steps: after a voice instruction sent by a user is received, performing text recognition on the voice instruction to obtain a matched instruction text; analyzing the instruction text based on a generative large model to obtain text content, and determining a video category instruction matched with the text content; obtaining a matched target category video according to the video category instruction, and obtaining a picture in the target category video; if image recognition features matched with the text content are screened out from the key features of the picture, reply content for the voice instruction is obtained according to the image recognition features and the text content; and generating an execution instruction of the vehicle based on the reply content, and controlling the vehicle to execute the execution instruction.

Description

Technical Field

[0001] The present application relates to the field of vehicle retrieval technology, and more specifically, to a speech interaction method and device based on a generative large model. Background Art

[0002] Setting up a voice assistant in a vehicle is one of the standard configurations of current vehicles. The functional adjustments of the vehicle are completed through information interaction between the voice assistant and the on-board terminal. In order to make the voice assistant more convenient for users to use, the generative big model is combined with the voice assistant.

[0003] At present, generative large models have been applied to more and more voice assistants in vehicles, but there is a lack of application scenarios and application content. For example, voice assistants can implement simple commands, such as adjusting the temperature of the air conditioner in the car and controlling the raising and lowering of the windows, by interacting with the vehicle terminal through traditional models. However, for more complex voice commands issued by users, such as obtaining images from a certain video stream in the vehicle, voice assistants cannot retrieve through traditional models, nor can they interact with the vehicle terminal. In addition, traditional models also have the defects of insufficient generalization and deduction capabilities and lack of context understanding capabilities, so they cannot perform complex vehicle control and play a corresponding role. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a voice interaction method and device based on a generative big model. This voice interaction method and device based on a generative big model effectively solves the problem that existing voice assistants cannot effectively recognize user voice commands through generative big models.

[0005] In a first aspect, an embodiment of the present application provides a speech interaction method based on a generative large model, which is applied to a vehicle-mounted terminal, and the method includes:

[0006] After receiving the voice command sent by the user, performing text recognition on the voice command to obtain a matching command text;

[0007] Analyze the instruction text based on the generative large model to obtain text content, and determine the video category instruction that matches the text content; obtain a matching target category video according to the video category instruction, and obtain a picture in the target category video; if an image recognition feature that matches the text content is screened out from the key features of the picture, then obtain a reply content for the voice instruction according to the image recognition feature and the text content;

[0008] An execution instruction of the vehicle is generated based on the reply content, and the vehicle is controlled to execute the execution instruction.

[0009] In combination with the first aspect, the embodiment of the present application provides a first possible implementation of the first aspect, wherein the determining the video category instruction matching the text content includes:

[0010] Extracting keywords from the text content and determining instruction tags for the keywords;

[0011] Based on the instruction tag, it is determined whether the text content matches the video category instruction; if so, the target category video is located based on the video category instruction matched by the instruction tag.

[0012] In combination with the first aspect, the embodiment of the present application provides a second possible implementation of the first aspect, wherein the step of acquiring a matching target category video according to the video category instruction includes:

[0013] Acquire a target category video matching the video category instruction from a plurality of category videos pre-stored in the vehicle; the target category video comprises at least one of an exterior video, an interior video, and a multimedia video;

[0014] or,

[0015] According to the video category instruction, real-time shooting is controlled to obtain a target category video; the target category video includes at least one of an outside vehicle video and an inside vehicle video.

[0016] In combination with the first aspect, the embodiment of the present application provides a third possible implementation of the first aspect, wherein the target category video includes an outside-vehicle video and an inside-vehicle video; and obtaining a picture in the target category video includes:

[0017] Selecting valid target category videos in a specified historical time period from the target category videos;

[0018] Extract the first key information from the text content, perform image positioning and image capture on the valid target category video according to the first key information, and obtain a corresponding picture; wherein the first key information corresponding to the video outside the vehicle and the video inside the vehicle is different.

[0019] In combination with the first aspect, the embodiment of the present application provides a fourth possible implementation of the first aspect, wherein the target category video includes a multimedia video; and obtaining a picture in the target category video includes:

[0020] Extracting second key information from the text content;

[0021] Image positioning and image capture are performed on the multimedia video according to the second key information to obtain a corresponding picture; wherein the second key information is different from the first key information.

[0022] In combination with the first aspect, the embodiment of the present application provides a fifth possible implementation of the first aspect, wherein if an image recognition feature matching the text content is screened out from the key features of the picture, a reply content for the voice command is obtained according to the image recognition feature and the text content, including:

[0023] If an image recognition feature matching the text content is selected from the key features of the image, a reference image matching the image recognition feature is retrieved from a preset database or the Internet; the reference image includes an image of complete information used to reply to the voice command;

[0024] A reply content for the voice command is obtained according to the image recognition feature, the picture feature of the reference picture and the text content.

[0025] In combination with the first aspect, the embodiment of the present application provides a sixth possible implementation of the first aspect, wherein the method further includes:

[0026] If there is no key image feature whose similarity with the keyword feature meets the preset threshold;

[0027] The voice assistant is controlled to broadcast the catch-all reply content generated by the generated large model according to the preset content rules.

[0028] In a second aspect, an embodiment of the present application provides a speech interaction method based on a generative large model, which is applied to a vehicle-mounted terminal, and the method includes:

[0029] A recognition module is used to perform text recognition on the voice command to obtain a matching command text after receiving the voice command sent by the user;

[0030] A screening module is used to analyze the instruction text based on the generative large model to obtain text content, and determine the video category instruction that matches the text content; obtain a matching target category video according to the video category instruction, and obtain a picture in the target category video; if an image recognition feature that matches the text content is screened out from the key features of the picture, then obtain a reply content for the voice instruction according to the image recognition feature and the text content;

[0031] An execution module is used to generate an execution instruction for the vehicle based on the reply content, and control the vehicle to execute the execution instruction.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, any step of a speech interaction method based on a generative large model is performed.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the computer program executes any one of the steps of a speech interaction method based on a generative large model.

[0034] A speech interaction method based on a generative large model is provided in an embodiment of the present application and is applied to a vehicle-mounted terminal. The method first performs text recognition on the speech instruction to obtain a matching instruction text after receiving a speech instruction sent by a user; then analyzes the instruction text based on the generative large model to obtain text content, and determines a video category instruction matching the text content; obtains a matching target category video according to the video category instruction, and obtains a picture in the target category video; if an image recognition feature matching the text content is screened out from the key features of the picture, a reply content for the speech instruction is obtained according to the image recognition feature and the text content; finally, an execution instruction of the vehicle is generated based on the reply content, and the vehicle is controlled to execute the execution instruction, which effectively solves the problem that the existing voice assistant cannot effectively recognize the user's voice instruction through the generative large model. The generative large model adopted in the present application makes up for the deficiencies of the traditional model in generalization and deduction capabilities and improves the context understanding ability, that is, the complexity of the speech instructions processed by the voice assistant through the generative large model is improved, and the accuracy of the control of the vehicle and the safety of the vehicle during driving are guaranteed, thereby ensuring the user's usage experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A schematic diagram of the process of a first speech interaction method based on a generative large model provided in an embodiment of the present application is shown;

[0037] Figure 2A schematic diagram of a process for locating target category videos provided in an embodiment of the present application is shown;

[0038] Figure 3 A schematic diagram of a process for obtaining a picture provided in an embodiment of the present application is shown;

[0039] Figure 4 A schematic diagram of the structure of a first speech interaction device based on a generative large model provided in an embodiment of the present application is shown;

[0040] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0042] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0043] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0044] At present, the generative big model has the problem of lack of application scenarios and application content. For example, the voice assistant can implement simple commands, such as adjusting the temperature of the air conditioner in the car and controlling the raising and lowering of the windows, by exchanging information with the vehicle terminal through the generative big model. However, for more complex voice commands issued by the user, such as obtaining images from a certain video stream on the vehicle, the voice assistant cannot retrieve through the generative big model, nor can it interact with the vehicle terminal. In addition, the traditional model also has the defects of insufficient generalization and deduction capabilities and lack of context understanding ability, so it cannot perform complex control of the vehicle and play a corresponding role.

[0045] Based on this, the embodiments of the present application provide a voice interaction method and device based on a generative large model, which will be described below through embodiments.

[0046] Example 1

[0047] To facilitate understanding of this embodiment, a speech interaction method based on a generative large model disclosed in an embodiment of this application is first described in detail. Figure 1 The flowchart of a speech interaction method based on a generative large model is shown in the figure, and the present application is applied to a vehicle terminal, and the method includes:

[0048] S101, after receiving a voice command sent by a user, performing text recognition on the voice command to obtain a matching command text;

[0049] S102, analyzing the instruction text based on the generative large model to obtain text content, and determining a video category instruction matching the text content; obtaining a matching target category video according to the video category instruction, and obtaining a picture in the target category video; if an image recognition feature matching the text content is screened out from key features of the picture, then obtaining a reply content for the voice instruction according to the image recognition feature and the text content;

[0050] S103: Generate an execution instruction for the vehicle based on the reply content, and control the vehicle to execute the execution instruction.

[0051] In the present application, the communication method between the voice assistant, the generative large model and the vehicle-mounted terminal is an inherent communication method on the vehicle, such as through a bus method, a wireless transmission method, or a wired transmission method.

[0052] In step S101, while the vehicle is driving, the user starts a voice conversation after waking up the voice assistant, and uses voice to speak voice commands to avoid affecting the safety of the vehicle while driving. The voice assistant collects the voice commands sent by the user through speakers, microphones, etc., and the vehicle-mounted terminal calls the recognition algorithm based on the acquired user's voice commands to automatically convert the voice commands into command text, that is, there is a content matching relationship between the voice commands and the command text. The recognition algorithm is a preset trained neural convolutional network that realizes the voice-to-text function. The specific training method is a mature existing technology and will not be elaborated here.

[0053] During the process of recognizing the voice command, a command text corresponding to the voice command is obtained, and during the recognition process, noise, redundant information or irrelevant words in the text command, such as filler words such as "um" and "ah", are removed to ensure the validity and purity of the command text, and the generated command text is verified to ensure that it complies with grammatical rules, is unambiguous, and matches the functions of the vehicle control system. If errors or inconsistencies are found, the user's voice command needs to be obtained again for re-correction, which is more convenient for the generative large model to recognize and process.

[0054] In step S102, after obtaining the instruction text, the instruction text corresponding to the voice instruction is further analyzed by the generative big model, that is, the instruction after the format conversion is semantically understood by the generative big model to obtain the text content, the text content is the user's intention and key information, etc. The keywords include object and direction. For example, if the user's voice instruction is "What is the model of the vehicle in front of my car", the text content is "Identify the model of the vehicle in front", and the keywords here are "identify", "front", "vehicle", and "model". To ensure accuracy, the generative big model can also verify the text content and voice instruction to ensure the accuracy of semantic understanding. The vehicle-mounted terminal can store the user's voice instruction and the text content for easy backtracking, and perform type matching based on the text content. If the text content matches the video category instruction, the generative big model locates the matching target category video according to the video category instruction, that is, the generative big model further determines the text content. The video category instruction that matches the specific target category video, corresponding measures are taken for the target category video to obtain the picture in the target category video, such as determining the required picture according to key information, the generated large model extracts key features from the picture, the key features of the picture can be in the form of pixels, or in the form of fluorescence, the key features of the image also include color, texture, shape, object position, scene, car logo and other categories, wherein the extraction method is to extract through a trained convolutional neural network, and the specific convolutional neural network training process is such as the prior art, which is not repeated here, if the generated large model screens out image recognition features that match the text content from the key features of the picture, that is, the image recognition features match the text content analyzed by the generated large model, and there is a corresponding relationship, then at this time the generated large model determines that the answer corresponding to the user's voice command is obtained, then the generated large model obtains the reply content for the voice command according to the image recognition features and the text content.

[0055] If the generative big model determines based on the text content that the user's voice command does not have a video category instruction, such as making a call to XXX, or sending a text message to XXX, opening a video playback software, generating AI drawings, etc., then the generative big model controls the vehicle-mounted terminal to perform corresponding actions according to the instructions that do not have video category instructions and generates corresponding reply content after the action is completed, and controls the voice assistant to play the reply content, such as controlling the voice assistant to play content such as making a call to XXX, or that the AI ​​drawing has been completed, etc.

[0056] In the specific implementation process of step S102, there is an embodiment: Figure 2 As shown, the step of determining the video category instruction matching the text content includes:

[0057] S10211, extracting keywords from the text content and determining instruction tags of the keywords;

[0058] S10212. Determine whether the text content matches a video category instruction based on the instruction tag; if so, locate the target category video based on the video category instruction matched by the instruction tag.

[0059] In steps S10211-S10212, after obtaining the text content, the generative large model extracts keywords in the text content according to preset extraction rules to realize the user's intention. The instruction label is usually a label associated with a specific category, which is assigned by the generative large model based on the text content. If the extracted keywords are "identification", "front", "vehicle", and "car model", the generative large model is mapped based on the above keywords, and the corresponding instruction label is determined to be "outside the car" based on the keywords, and based on this instruction label, it is judged whether the text content matches the video category instruction. The video category instructions include outside the car, inside the car, and multimedia. Among them, multimedia is multimedia that plays video pictures. At this time, the indicator label "outside the car" corresponds to the outside car video category instruction, and different video category instructions correspond to different categories of videos. The video located by the video category instruction matching the instruction label is the target category video, and the generative large model needs to obtain the picture corresponding to the text content from the target category video.

[0060] In the specific implementation process of step S10212, there is an embodiment in which: the step of obtaining a matching target category video according to the video category instruction includes:

[0061] S102121. Acquire a target category video that matches the video category instruction from a plurality of category videos pre-stored in the vehicle; the target category video includes at least one of an exterior video, an interior video, and a multimedia video;

[0062] S102122, or,

[0063] According to the video category instruction, real-time shooting is controlled to obtain a target category video; the target category video includes at least one of an outside vehicle video and an inside vehicle video.

[0064] In steps S102121-S102122, the generative large model calls the video resources in the vehicle. The video resources are collected in various ways, such as collected by multiple cameras inside and outside the vehicle or generated during the operation of multimedia. Then, the video category of each of the multiple video resources in the vehicle is classified according to the preset classification rules to obtain multiple video categories. In the present application, the preset classification rules are that the locations of the collected videos are inside the vehicle, outside the vehicle and multimedia respectively. For example, the video resources generated by the multimedia are cached during the operation of the multimedia. For example, the video outside the vehicle is obtained by the vehicle terminal by cyclically caching the camera videos in the four directions of the front, back, left and right of the vehicle at the same time, with a duration of 20s, which is stored in a designated directory of the vehicle terminal for calling when used. Then the video categories include but are not limited to inside the vehicle, outside the vehicle and multimedia. Then, based on the instruction label obtained from the text content in the generative large model, the target category video matching the video category instruction is located from the multiple video categories. If the instruction label is "outside the vehicle", the target category video matching the video category instruction is located as the outside vehicle video based on the instruction label "outside the vehicle";

[0065] When the target category video is an in-car video or an in-car video, the corresponding target category video can also be obtained by real-time shooting by controlling the corresponding camera or other components with the function of taking pictures based on the video category instructions shown. For example, in the target category video of the out-of-car video, the user's instructions include the front, then real-time shooting of the front is required, but since the multimedia video is intercepted when it is played in real time on the display, there is no need to shoot.

[0066] In the specific implementation process of step S102, there is another embodiment: Figure 3 As shown, the target category video includes an outside-vehicle video and an inside-vehicle video; the step of obtaining a picture in the target category video includes:

[0067] S10221. Select valid target category videos in a specified historical time period from the target category videos;

[0068] S10222. Extract the first key information from the text content, and perform image positioning and image capture on the valid target category video according to the first key information to obtain a corresponding picture; wherein the first key information corresponding to the video outside the vehicle and the video inside the vehicle is different.

[0069] In steps S10221-S10222, when the generative large model determines that the video type instruction corresponding to the instruction label is an outdoor video or an indoor video, it is necessary to select a valid target category video in a specified historical time period from the target category video, that is, to obtain the outdoor video when the video is outside the vehicle, and select the valid target category video in the specified historical time period, instead of selecting all the outdoor videos. At this time, the generative large model extracts the first key information based on the text content, wherein the first key information corresponding to the outdoor video is different from the first key information corresponding to the indoor video, and the first key information corresponding to the outdoor video includes the object, direction and the specified historical time period. For example, the user's voice command is to open the image of the indoor camera at noon on March 5, 2024. Since noon is a time period, the positioning obtains a video clip; and the first key information corresponding to the indoor video includes the object, that is, the vehicle hand part, the instruction historical time period but no direction exists. For example, the user's voice command is "the indoor instrument What does the red warning displayed on the dial mean?" The keywords are instrument panel and red warning, where the instrument panel is the vehicle opponent, and the opponent in the car can also be a display, a door, a window, etc.; based on the first key information, the image positioning and image capture of the valid target category video are performed, and only one picture is captured in this setting, and the text content can also be associated with a specific event (such as collision detection, pedestrian detection, etc.). The generative large model will locate the video clip according to the tag or sensor data of the event, and in order to obtain an image from the video clip, the image is captured from the video clip. The image capture method is preset, or it can be recognized according to the user's voice command, such as single frame capture or key frame capture, so as to obtain the picture corresponding to the text content. The generative large model can also process the picture, such as denoising, contrast enhancement, color adjustment, etc., to improve the image quality or make it more suitable for the application scenario specified by the user, such as license plate recognition, face detection, etc.

[0070] In the specific implementation process of step S102, there is another embodiment in which: the target category video includes a multimedia video; and obtaining a picture in the target category video includes:

[0071] S10231, extracting second key information from the text content;

[0072] S10233. Perform image positioning and image capture on the multimedia video according to the second key information to obtain a corresponding picture; wherein the second key information is different from the first key information.

[0073] In this step S10231-S10232, when the generative large model determines that the video type instruction corresponding to the instruction tag is a multimedia category, the corresponding target category video is a multimedia video, that is, the user's voice instruction is similar to "What is the name of the supporting role in the picture being played now?", then the generative large model extracts the corresponding second key information according to the text content, and the second key information includes verbs, nouns, specific time segments, etc. in the video picture, characters or text content, that is, the second key information is quite different from the first key information, and the second key information contains more details. The second key information is used to locate the screen of TV, movies, short videos, etc. played on the display screen of the vehicle-mounted terminal, and the screen is captured based on the located screen to obtain the corresponding picture.

[0074] In the specific implementation process of step S102, there is another embodiment: if the image recognition feature matching the text content is selected from the key features of the picture, then the reply content for the voice command is obtained according to the image recognition feature and the text content, including:

[0075] S10241. If an image recognition feature matching the text content is selected from the key features of the image, a reference image matching the image recognition feature is retrieved from a preset database or the Internet; the reference image includes an image of complete information used to reply to the voice command;

[0076] S10242. Obtain reply content for the voice command according to the image recognition feature, the picture feature of the reference picture, and the text content.

[0077] In steps S10241-S10242, after obtaining the corresponding picture, the generative large model extracts the key features of the picture from the picture according to a preset extraction method, the number of the key features is at least one, and matches the key features with the key information in the text content. If they match, the extracted key features are image recognition features, that is, there is information corresponding to the text content in the picture. For example, if the user's voice command is "What is the model of the vehicle in front of my car", and the text content is "Identify the model of the vehicle in front", then at least the key feature of "car" must exist in the picture, and "car" is used as the image recognition feature. After obtaining the image recognition feature, the generative large model retrieves a reference picture that matches the image recognition feature in a preset database or on the Internet, the reference picture includes a picture of complete information used to reply to the voice command, and the reference picture also contains all information for identifying the model of the car in the picture; that is, The generative big model compares the similarity of the picture, the reference picture and the text content, that is, compares the similarity of the image recognition features of the picture, the key picture features of the reference picture and the key information between the text content. If the three correspond one to one, that is, there is a key picture feature in the reference picture corresponding to the image recognition feature in the picture, that is, the similarity between the two meets the threshold, wherein the threshold is specifically set based on the actual situation, then the generative big model determines that the answer corresponding to the user's voice command is obtained. At this time, the generative big model generates the corresponding reply content based on the answer using the corresponding preset content rules, wherein the preset content rules can be that the generative big model parameterizes the image recognition features of the picture, the key picture features of the reference picture and the key information between the text content, thereby generating the corresponding reply content. The preset database is pre-established by the vehicle-mounted terminal, and the Internet is the function of the vehicle-mounted terminal to access the Internet.

[0078] In the multimedia video category, the character picture in the user's voice command is intercepted and the character is identified to determine whether it corresponds to the character or animal in the user's voice command, and a preset database or the Internet is searched through the character picture to obtain the actor information corresponding to the character, and the key information in the text content is verified and confirmed again, and the corresponding reply content is generated after the verification and confirmation are passed, and the voice assistant is controlled to play the reply content to reply to the user, thereby achieving the effect of interacting with the user around the played content, thereby further ensuring the accuracy and precision of the generative large model when replying to the user.

[0079] In the in-car video category, if the user's voice command is "What does the red warning displayed on the dashboard mean?", the generative large model obtains the image recognition features based on the picture, and searches the image recognition features in a preset database or the Internet, and verifies and confirms them with the key information in the text content, and calls the on-board terminal to analyze the instruments corresponding to the keywords, and generates reply content, such as "The vehicle is low on fuel" or "The door is not closed", then the generative large model generates reply content to the user based on the feedback results, and controls the voice assistant to play the reply content.

[0080] In the category of out-of-car videos, if the user's voice command is "What is the model of the vehicle in front of my car?", the generative large model obtains the image recognition features based on the picture, searches the image recognition features in a preset database or the Internet, and verifies and confirms them with the key information in the text content. When the generative large model determines the model corresponding to the vehicle in front, it controls the voice assistant to broadcast "The model of the vehicle in front is XXXX".

[0081] In the specific implementation process of step S102, there is also an embodiment: the method further includes:

[0082] S10251: If there is no key image feature whose similarity with the keyword feature meets a preset threshold;

[0083] S10252, controlling the voice assistant to broadcast the catch-all reply content generated by the generated large model according to the preset content rules.

[0084] In steps S10251-S10252, the generative big model compares the similarity of the picture, the reference picture and the text content, that is, compares the image recognition features of the picture, the key picture features of the reference picture and the key information between the text content. If it is determined that there is no key picture feature in the reference picture whose similarity with the keyword feature meets the preset threshold, that is, there is no corresponding relationship between the reference picture and the picture, then there is no corresponding relationship between the key information of the text content. At this time, the generative big model adopts the corresponding preset content rules to generate a catch-all reply content, and controls the voice assistant to broadcast the catch-all reply content to form a reply to the user's voice command. For example, the user's voice command is "What is the model of the vehicle in front of my car?" When the generative big model determines that there is no vehicle in front or the corresponding model is not found, it controls the voice assistant to broadcast "The vehicle in front has not been identified yet" or "The model of the vehicle in front has not been identified yet."

[0085] In step S103, after the generative large model generates the reply content, the vehicle-mounted terminal analyzes the reply content. If the reply content contains a control requirement for the vehicle, such as the user's voice command is "It's so cold in the car", and the reply content generated by the generative large model is "The vehicle's air conditioning and hot air need to be turned on", then an execution instruction for the vehicle is generated based on the reply content, and the execution instruction is "Turn on the vehicle's air conditioning and hot air", and the vehicle is controlled to execute the execution instruction, thereby increasing the temperature of the vehicle to respond to the user in a targeted manner. If there is no control requirement for the vehicle in the reply content, the vehicle is not controlled, thereby forming intelligent control of the vehicle.

[0086] Example 2

[0087] The present application also provides a speech interaction device based on a generative large model, such as Figure 4 The block diagram of a speech interaction device based on a generative large model is shown. The function implemented by the speech interaction device based on a generative large model corresponds to the steps of executing a speech interaction method based on a generative large model on a terminal device. The device can be understood as a component of a server including a processor, which is applied to a vehicle-mounted terminal. The method includes:

[0088] The recognition module 401 is used to perform text recognition on the voice command to obtain a matching command text after receiving the voice command sent by the user;

[0089] The screening module 402 is used to analyze the instruction text based on the generative large model to obtain text content, and determine the video category instruction matching the text content; obtain the matching target category video according to the video category instruction, and obtain the picture in the target category video; if the image recognition feature matching the text content is screened out from the key features of the picture, then obtain the reply content for the voice instruction according to the image recognition feature and the text content;

[0090] The execution module 403 is used to generate an execution instruction for the vehicle based on the reply content, and control the vehicle to execute the execution instruction.

[0091] In a feasible implementation, the screening module includes:

[0092] A first extraction module, used to extract keywords from the text content and determine instruction tags of the keywords;

[0093] The judging module is used to judge whether the text content matches the video category instruction based on the instruction tag; if so, locate the target category video based on the video category instruction matched by the instruction tag.

[0094] In a feasible implementation, the screening module also includes:

[0095] An acquisition module, configured to acquire a target category video matching the video category instruction from a plurality of category videos pre-stored in the vehicle; the target category video comprises at least one of an exterior video, an interior video, and a multimedia video;

[0096] or,

[0097] The shooting module is used to control real-time shooting and obtain a target category video according to the video category instruction; the target category video includes at least one of an outside vehicle video and an inside vehicle video.

[0098] In a feasible implementation, the screening module further includes:

[0099] A selection module, used to select valid target category videos in a specified historical time period from the target category videos;

[0100] The second extraction module is used to extract the first key information in the text content, and perform image positioning and image capture on the valid target category video according to the first key information to obtain a corresponding picture; wherein the first key information corresponding to the outside-vehicle video and the inside-vehicle video is different.

[0101] In a feasible implementation, the screening module further includes:

[0102] A second extraction module, used to extract second key information from the text content;

[0103] A positioning module is used to perform image positioning and image capture on the multimedia video according to the second key information to obtain a corresponding picture; wherein the second key information is different from the first key information.

[0104] In a feasible implementation manner, the screening module further includes:

[0105] A retrieval module, for retrieving a reference image matching the image recognition feature from a preset database or the Internet if an image recognition feature matching the text content is selected from the key features of the image; the reference image includes an image of complete information for replying to the voice command;

[0106] The reply module is used to obtain reply content for the voice instruction according to the image recognition feature, the picture feature of the reference picture and the text content.

[0107] In a feasible implementation, the screening module further includes:

[0108] A satisfying module, used for checking if there is no key image feature whose similarity with the keyword feature meets a preset threshold;

[0109] The generation module is used to control the voice assistant to broadcast the catch-all reply content generated by the generation model according to the preset content rules.

[0110] Example 3

[0111] The present application also provides an electronic device, such as Figure 5 As shown, it includes: a processor 501, a memory 502 and a bus 503, the memory 502 stores machine-readable instructions executable by the processor 501, when the electronic device is running, the processor 501 and the memory 502 communicate through the bus 503, and when the machine-readable instructions are executed by the processor 501, any step of the speech interaction method based on the generative large model is performed.

[0112] Example 4

[0113] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes any one of the steps of the speech interaction method based on a generative large model.

[0114] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0115] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0117] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a platform server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0118] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A speech interaction method based on a generative large model, characterized in that: Applied to a vehicle-mounted terminal, the method comprises: After receiving the voice command sent by the user, performing text recognition on the voice command to obtain a matching command text; Analyze the instruction text based on the generative large model to obtain text content, and determine the video category instruction that matches the text content; obtain a matching target category video according to the video category instruction, and obtain a picture in the target category video; if an image recognition feature that matches the text content is screened out from the key features of the picture, then obtain a reply content for the voice instruction according to the image recognition feature and the text content; An execution instruction of the vehicle is generated based on the reply content, and the vehicle is controlled to execute the execution instruction.

2. The method according to claim 1, characterized in that The determining of the video category instruction matching the text content includes: Extracting keywords from the text content and determining instruction tags for the keywords; Based on the instruction tag, it is determined whether the text content matches the video category instruction; if so, the target category video is located based on the video category instruction matched by the instruction tag.

3. The method according to claim 2, characterized in that The step of obtaining a matching target category video according to the video category instruction includes: Acquire a target category video matching the video category instruction from a plurality of category videos pre-stored in the vehicle; the target category video comprises at least one of an exterior video, an interior video, and a multimedia video; or, According to the video category instruction, real-time shooting is controlled to obtain a target category video; the target category video includes at least one of an outside vehicle video and an inside vehicle video.

4. The method according to claim 1 or 2, characterized in that: The target category video includes an outside-vehicle video and an inside-vehicle video; and obtaining a picture in the target category video includes: Selecting valid target category videos in a specified historical time period from the target category videos; Extract the first key information from the text content, perform image positioning and image capture on the valid target category video according to the first key information, and obtain a corresponding picture; wherein the first key information corresponding to the video outside the vehicle and the video inside the vehicle is different.

5. The method according to claim 1 or 2, characterized in that: The target category video includes a multimedia video; and obtaining a picture in the target category video includes: Extracting second key information from the text content; Image positioning and image capture are performed on the multimedia video according to the second key information to obtain a corresponding picture; wherein the second key information is different from the first key information.

6. The method according to claim 1, characterized in that If an image recognition feature matching the text content is selected from the key features of the picture, then a reply content for the voice command is obtained according to the image recognition feature and the text content, including: If an image recognition feature matching the text content is selected from the key features of the image, a reference image matching the image recognition feature is retrieved from a preset database or the Internet; the reference image includes an image of complete information used to reply to the voice command; A reply content for the voice command is obtained according to the image recognition feature, the picture feature of the reference picture and the text content.

7. The method according to claim 1, characterized in that The method further comprises: If there is no key image feature whose similarity with the keyword feature meets the preset threshold; The voice assistant is controlled to broadcast the catch-all reply content generated by the generated large model according to the preset content rules.

8. A speech interaction method based on a generative large model, characterized in that: Applied to a vehicle-mounted terminal, the method comprises: A recognition module is used to perform text recognition on the voice command to obtain a matching command text after receiving the voice command sent by the user; A screening module is used to analyze the instruction text based on the generative large model to obtain text content, and determine the video category instruction that matches the text content; obtain a matching target category video according to the video category instruction, and obtain a picture in the target category video; if an image recognition feature that matches the text content is screened out from the key features of the picture, then obtain a reply content for the voice instruction according to the image recognition feature and the text content; An execution module is used to generate an execution instruction for the vehicle based on the reply content, and control the vehicle to execute the execution instruction.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of a speech interaction method based on a generative large model are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of a speech interaction method based on a generative large model as described in any one of claims 1 to 7.

Images