Video interaction method and device based on large model, electronic equipment and medium

Abstract

The invention provides a video interaction method and device based on a large model, electronic equipment and a medium, and relates to the technical field of artificial intelligence, in particular to the technical field of large models, natural language processing and image processing. According to the implementation scheme, in the video interaction process based on the large model, input interaction input data and a target file are obtained, and the interaction input data comprise any one of voice data, image data and video data; processing the interactive input data and the target file through a large model to determine interactive output data; and outputting the interactive output data.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of large models, natural language processing and image processing technology, and specifically to a video interaction method, device, electronic device, computer-readable storage medium and computer program product based on a large model. Background Art

[0002] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] A large model (also called a foundation model) refers to a machine learning model with a large number of parameters and complex structure. It can process massive amounts of data and complete various complex tasks, such as natural language processing, computer vision, and speech recognition.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides a large model-based video interaction method, device, electronic device, computer-readable storage medium and computer program product.

[0006] According to one aspect of the present disclosure, a video interaction method based on a big model is provided, comprising: in a video interaction process based on a big model, obtaining input interaction input data and a target file, wherein the interaction input data includes any one of voice data, image data and video data; processing the interaction input data and the target file through the big model to determine interaction output data; and outputting the interaction output data.

[0007] According to another aspect of the present disclosure, a video interaction device based on a big model is provided, comprising: a first module, configured to obtain input interaction input data and a target file during a video interaction process based on a big model, wherein the interaction input data comprises any one of voice data, image data and video data; a second module, processing the interaction input data and the target file through the big model to determine interaction output data; and a third module, configured to output the interaction output data.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0011] According to one or more embodiments of the present disclosure, a video interaction method based on a large model is provided. By obtaining target files other than interactive input data (e.g., real-time video streams or real-time audio streams, etc.) as supplementary information during the video interaction process, the large model can be assisted to better understand the current interactive intent of the interactive object, effectively improving the accuracy of the large model's understanding of the interactive intent of the interactive object, thereby improving the matching degree between the output information and the interactive intent, and effectively improving the accuracy of the large model's understanding of the interactive intent during the video interaction process. Moreover, compared with real-time interactive input data, the target file has relatively low requirements for processing power, and usually does not occupy too much computing resources of the large model, which can ensure the normal operation of the video interaction function.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 is a schematic diagram illustrating an example system in which the various methods described herein may be implemented according to an exemplary embodiment;

[0015] Figure 2 A flowchart of a video interaction method based on a large model according to an embodiment of the present disclosure is shown;

[0016] Figure 3 A partial flow chart of another large model-based video interaction method according to an embodiment of the present disclosure is shown;

[0017] Figure 4 A partial flow chart of another large model-based video interaction method according to an embodiment of the present disclosure is shown;

[0018] Figure 5 A structural block diagram of a video interaction device based on a large model according to an embodiment of the present disclosure is shown; and

[0019] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0020] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0021] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0022] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations.

[0023] In the related art, a large model can be used to process the real-time video stream collected from users who are interacting with the video, so as to understand the user's current interaction intention, thereby realizing the real-time video interaction function between artificial intelligence and the user. However, the computational complexity of processing the real-time video stream is high, and the video interaction effect achieved solely based on the real-time video stream is not ideal.

[0024] To solve the above problems, the present disclosure provides a video interaction method based on a big model. By obtaining target files other than interaction input data (for example, real-time video streams or real-time audio streams, etc.) as supplementary information during the video interaction process, the big model can be assisted to better understand the current interaction intentions of the interaction objects, and the accuracy of the big model's understanding of the interaction intentions of the interaction objects can be effectively improved, thereby improving the matching degree between the output information and the interaction intentions, and effectively improving the accuracy of the big model's understanding of the interaction intentions during the video interaction process.

[0025] Moreover, compared with real-time interactive input data, the target file has relatively low requirements for processing power and usually does not occupy too much computing resources of the large model, thus ensuring the normal operation of the video interaction function.

[0026] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0027] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0028] In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable the large model-based video interaction method to be performed.

[0029] In some embodiments, server 120 may also provide other services or software applications that may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0030] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is an example of a system for implementing the methods described herein and is not intended to be limiting.

[0031] The user may use client devices 101, 102, 103, 104, 105 and / or 106 to perform the large model-based video interaction method. The client device may provide an interface that enables the user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0032] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0033] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0034] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0035] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0036] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0037] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0038] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0039] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0040] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0041] Figure 2 A flowchart of a large model-based video interaction method according to an embodiment of the present disclosure is shown.

[0042] like Figure 2 As shown, the video interaction method 200 based on the large model includes:

[0043] Step 210: in the video interaction process based on the large model, obtaining input interactive input data and a target file, wherein the interactive input data includes any one of voice data, image data and video data;

[0044] Step 220: Process the interactive input data and the target file through the large model to determine interactive output data; and

[0045] Step 230: Output the interactive output data.

[0046] Therefore, by obtaining target files other than interactive input data (for example, real-time video streams or real-time audio streams, etc.) as supplementary information during the video interaction process, the big model can be assisted to better understand the current interaction intentions of the interactive objects, and effectively improve the accuracy of the big model's understanding of the interaction intentions of the interactive objects, thereby improving the matching degree between the output information and the interaction intentions, and effectively improving the accuracy of the big model's understanding of the interaction intentions during the video interaction process.

[0047] Moreover, compared with real-time interactive input data, the target file has relatively low requirements for processing power and usually does not occupy too much computing resources of the large model, thus ensuring the normal operation of the video interaction function.

[0048] In step 210, the interaction input data may be, for example, a screen image, a real-time interaction audio stream, and a real-time interaction video stream collected by an interaction object side device (e.g., a camera on the user side, etc.), thereby enabling the large model to understand the current interaction intention of the interaction object based on technologies such as speech recognition, image processing, and natural language processing.

[0049] Exemplarily, the interactive objects may be, for example, users and other large models, etc., which are not limited.

[0050] Exemplarily, the video interaction may be implemented based on at least one of a video conversation function and a screen sharing function.

[0051] In the example, when the video interaction is a video conversation, the big model determines the language, expression, body movements, and environment of the interaction object through the real-time interaction video stream to determine the current interaction intention of the interaction object.

[0052] In the example, when the video interaction is screen sharing, the large model determines the image and text content in the interactive object's screen through the real-time interactive video stream, and can determine the interactive object's current interaction intention in combination with the interactive object's voice audio.

[0053] In the example, the video conversation and screen sharing can also be performed simultaneously or interactively. For example, when screen sharing is performed, a small window of the video conversation function can be retained on the screen to obtain relevant information such as the expression and environment of the interactive object from the small window.

[0054] According to some embodiments, method 200 also includes: step 240, in response to determining that it is in a preset situation, outputting a file upload prompt, wherein the preset situation indicates that the large model is adversely affected when processing the current interaction intention of the video interaction; and step 210 at least includes: in response to detecting a file upload operation, obtaining an input target file.

[0055] Processing real-time video streams requires a large amount of network bandwidth and computing resources, and has high requirements on the condition of the collected interactive video streams (for example, clarity and completeness). Therefore, in situations such as poor network conditions or poor conditions of the collected interactive video streams, it is easy for the large model to have deviations in its understanding of user intentions.

[0056] Therefore, when an adverse impact on the big model's understanding of the current interaction intention is detected, a prompt to upload the target file can be generated to further obtain, for example, the target file uploaded by the interactive object to assist the big model in understanding, so as to improve the accuracy of the big model's intention understanding, and thereby improve the matching degree between the interaction output data and the intention of the interactive object.

[0057] According to some embodiments, the adverse impact is associated with at least one of the following: a network condition on the large model side, a network condition on the interactive object side that interacts with the large model in video interaction, and a condition of interactive input data.

[0058] According to some embodiments, in the case where the adverse effect is associated with the network condition on the large model side, the preset situation at least includes that the network transmission rate of the large model is lower than a first threshold; and wherein, in the case where the adverse effect is associated with the network condition on the interactive object side, the preset situation at least includes that the network transmission rate of the interactive object is lower than a second threshold.

[0059] The processing of large models is highly dependent on network transmission rate. Specifically, when the network transmission rate of either side of the video interaction is poor, it is easy to cause delays in the large model receiving, calculating and sending data, seriously affecting the real-time nature of the video interaction. By obtaining the target file when adverse effects related to network conditions are detected, the smooth implementation of video interaction can be guaranteed as much as possible at the expense of a small amount of computing resources.

[0060] Exemplarily, the first threshold and the second threshold may be the same or different.

[0061] Exemplarily, the first threshold and the second threshold may be determined according to the complexity of the intention of the interactive object. For example, in response to the large model determining that the intention of the interactive object is relatively complex, the first threshold and the second threshold may be set relatively high to more timely determine and obtain the uploaded target file.

[0062] Exemplarily, the first threshold and the second threshold may also be determined according to the performance of the large model. For example, a large model with a complex structure generally requires a higher network transmission rate than a large model with a simple structure, so the threshold set for it may be higher than that for a simple large model.

[0063] According to some embodiments, in response to determining that the adverse effect is associated with the status of the interactive input data, the preset situation includes at least one of the following:

[0064] In the case where the interactive input data is image data or video data, the clarity of the interactive input data is lower than a third threshold, and

[0065] In the case where the interactive input data is voice data or video data, the interactive input data lacks data frames.

[0066] The intention understanding of the big model is also closely related to the status of the interactive input data. When the acquired real-time video stream is not clear enough, it is difficult for the big model to accurately determine the interactive object's interactive intention from the real-time video stream. Therefore, in this case, the input target file can be obtained as supplementary information to assist the big model in understanding, so as to improve the accuracy of the big model's intention understanding, and thus improve the matching degree between the interactive output data and the intention of the interactive object.

[0067] Exemplarily, the third threshold may be determined according to the complexity of the intention of the interactive object. For example, in response to the large model determining that the intention of the interactive object is relatively complex, the third threshold may be set relatively high to determine and obtain the uploaded target file more timely.

[0068] For example, when the interactive data is audio data or video data, since the data flow of the real-time interactive data stream is large and the total amount of data is not fixed, it is very easy to lose frames or omit frames during the data transmission process. Based on this, by setting the input target file to supplement the information when such a situation is detected, the accuracy of the large model in understanding the intention of the interactive object is effectively improved, and the overall effect of the video interaction is improved.

[0069] Exemplarily, a frame number threshold may be further set, and when the number of lost or missed frames exceeds the frame number threshold, the input target file is acquired to improve processing efficiency and reduce resource waste.

[0070] It should be noted that the above-mentioned adverse effects may also include, for example, insufficient network bandwidth for data transmission between the interactive object and the large model, damage to the camera of the interactive object and inability to collect interactive input data, etc., which are not limited to this.

[0071] In the example, the above method 200 can be used in highly professional scenarios such as conferences, academic discussions, and course teaching. In step 220, the target file can be, for example, literature, slides, and film and television materials related to professional topics prepared in advance, so as to provide better assistance for the large model to assist in understanding the intention of the interactive object.

[0072] For example, for the meeting scenario of the topic of "Summary of Company A's Robot Project", when the network transmission rate of the interactive object is lower than the first threshold or the network transmission rate of the large model is lower than the second threshold, the large model only receives the "Summary of Company A" part of the complete instruction "Summary of the main features of Company A's latest robot" issued by the interactive object. At this time, a button or dialog box with a "file upload" mark, for example, can be displayed on the interface of the interactive object side, so as to prompt the interactive object to upload target files such as the manual of Company A's latest robot, the promotional video of Company A's latest robot (or its website), etc., to assist the large model in understanding the current interactive intention of the interactive object.

[0073] Exemplarily, the number of interactive objects in the above conference scenario may also be greater than 1. For a multi-person conference scenario where the number of interactions is greater than 1, some or all interactive objects may be prompted to upload the target file. Furthermore, when, for example, the network transmission rate of some interactive objects is lower than the first threshold, only the interactive objects with poor network conditions may be prompted to upload the target file.

[0074] It should be noted that, in addition to the above-mentioned highly professional scenarios, the above-mentioned method 200 can also be applied to non-professional scenarios such as daily communication.

[0075] For example, when the interaction object is a user and the user intends to share emotions with the big model, the big model can usually determine the user's specific emotions based on the user's facial expressions and body movements, and make an emotional response. However, when the clarity of the collected real-time interactive video stream is lower than the threshold, it may be difficult for the big model to understand the user's specific emotions only by tone. At this time, a button or dialog box with a label such as "file upload" can be displayed in the user interface, allowing the user to upload target files such as articles, music or videos related to the current emotional state, thereby effectively assisting the big model to understand the user's current interaction intention (for example, the big model is required to provide a response with comforting emotions, etc.).

[0076] It should be understood that the above examples are for illustration purposes only and are not intended to be limiting.

[0077] According to some embodiments, the target file includes at least one of a text file, an audio file, an image file, and a video file. Exemplarily, the target file can be a file of various formats, for example, a txt format file (text file), a wma format file (audio file), a jpg format file (image file), and an mpeg format file (video file).

[0078] Figure 3 A partial flow chart of another large model-based video interaction method according to an embodiment of the present disclosure is shown.

[0079] According to some embodiments, Figure 3 As shown, in response to the target file including a video file, step 220 includes:

[0080] Step 310: Process the interaction input data through the large model to determine the initial interaction intention;

[0081] Step 320: Decode the video file to obtain multiple decoded video frames;

[0082] Step 330: determining a key frame associated with the initial interaction intention from a plurality of decoded video frames; and

[0083] Step 340: Determine the interaction output data based on the key frame and the initial interaction intention.

[0084] In this way, the large model can combine streaming audio and video data (interaction data) and complete video data (target file) to understand user intentions. Compared with streaming audio and video data or complete video data alone, the accuracy of the large model's intention understanding can be effectively improved.

[0085] Specifically, the processing of streaming video data is highly real-time, and the large model can only understand the current frame information and its previous information in the streaming audio and video data. Therefore, it is difficult to identify the current interaction intention of the interactive object based solely on streaming video data. In addition, the appearance of preset situations will further reduce the integrity of its current frame information and previous information, further increasing the difficulty of understanding.

[0086] Therefore, by introducing the target file of the complete video data type, the missing information can be effectively supplemented. And because both are video data, the processing flow is relatively similar. On the basis of simplifying the processing complexity as much as possible, it can provide more complete information for the large model to assist in intent understanding.

[0087] Figure 4A partial flow chart of another large model-based video interaction method according to an embodiment of the present disclosure is shown.

[0088] According to some embodiments, Figure 4 As shown, in addition to the above steps 210 to 230, the method 200 further includes:

[0089] Step 410: in response to detecting the interest information of the interactive object having the video interaction, acquiring the interest information; and

[0090] Step 420: Process the interest information, interactive input data and target file through the large model to determine interactive output data.

[0091] Therefore, by introducing the interest information of the interactive object, it is possible to predict in advance the content that the interactive object may need to interact with while occupying a small amount of computing resources, so as to better assist the large model in understanding the current interaction intention of the interactive object.

[0092] According to some embodiments, the topic of interest of the interactive object can be determined as the above-mentioned interest information by processing the historical interaction data (e.g., text conversation data, voice conversation data, and video conversation data, etc.) between the big model and the interactive object. For example, the historical interaction data between the big model and the user can be simplified into at least one first tag as the user's interest information, so as to assist the big model in understanding its current interaction intention based on the topic that the user may be interested in.

[0093] For example, in a historical conversation, if a user asks the big model "Twenty ways to cook vegetables", the "health" and "cooking" tags can be added to the user accordingly to indicate that the user has the possibility of interacting with related content based on the above two tags.

[0094] According to some embodiments, each first tag can be further associated with the specific time when the user pays attention to each topic of interest (for example, the time when the corresponding historical interaction behavior that generated the first tag occurred), so as to better analyze the changes in the user's topics of interest over time, so as to better assist the large model in understanding the user's current interaction intentions.

[0095] For example, in the first historical conversation, the user asked the big model "Twenty ways to cook vegetables", so the "health" and "cooking" tags are added to the user this time; and in the second historical conversation, the user asked the big model "good historical novels", so the "history" and "reading" tags are added to the user this time. Since the second historical conversation is closer to the current moment, the possibility of the user's current interaction intention being associated with the "history" and "reading" tags can be increased compared to the "health" and "cooking" tags.

[0096] According to some embodiments, in addition to the above steps 210 to 230, the method 200 further includes:

[0097] Step 250: output prompt information, where the prompt information is used to indicate the upload target file.

[0098] Exemplarily, the prompt information may be, for example, "The current network condition is not good, please upload files related to the meeting topic" or the like.

[0099] Exemplarily, a time threshold may be set, and when the uploaded target file is not obtained within the time threshold, a prompt message is output to prompt the interactive object to upload the target file, so as to ensure that the large model can continue to accurately support the video interaction function.

[0100] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0101] According to another aspect of the present disclosure, a video interaction device based on a large model is provided. Figure 5 As shown, the video interaction device 500 based on the big model includes: a first module 510, configured to obtain input interactive input data and target files during the video interaction process based on the big model, wherein the interactive input data includes any one of voice data, image data and video data; a second module 520, configured to process the interactive input data and the target file through the big model to determine the interactive output data; and a third module 530, configured to output the interactive output data.

[0102] According to another aspect of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the aforementioned method.

[0103] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable the computer to execute the aforementioned method.

[0104] According to another aspect of the present disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements the aforementioned method when executed by a processor.

[0105] like Figure 6As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0106] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608 and a communication unit 609. The input unit 606 can be any type of device that can input information to the electronic device 600. The input unit 606 can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a trackpad, a trackball, a joystick, a microphone and / or a remote control. The output unit 607 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator and / or a printer. The storage unit 608 can include but is not limited to a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver and / or a chipset, such as Bluetooth TM devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0107] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a matrix calculation method based on a GPU. For example, in some embodiments, a matrix calculation method based on a GPU may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via a ROM 602 and / or a communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the matrix calculation method based on the GPU described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform a matrix calculation method based on a GPU in any other appropriate manner (e.g., by means of firmware).

[0108] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0110] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0112] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0113] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0114] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0115] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but only by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples can be omitted or replaced by their equivalent elements. In addition, each step can be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples can be combined in various ways. It is important that with the evolution of technology, many elements described herein can be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A video interaction method based on a large model, comprising: In the video interaction process based on the large model, input interactive input data and a target file are obtained, wherein the interactive input data includes any one of voice data, image data and video data; Processing the interactive input data and the target file through the large model to determine interactive output data; and The interactive output data is output.

2. The method according to claim 1, further comprising: In response to determining that the situation is under a preset condition, outputting a file upload prompt, wherein the preset condition indicates that the large model is adversely affected when processing the current interaction intention of the video interaction; and In the video interaction process based on the large model, obtaining the input interactive input data and the target file at least includes: In response to detecting a file upload operation, the input target file is acquired.

3. The method according to claim 1 or 2, wherein: The target file includes at least one of a text file, an audio file, an image file, and a video file.

4. The method according to claim 3, wherein: In the case where the target file includes a video file, the step of processing the interactive input data and the target file by the large model to determine the interactive output data includes: Processing the interaction input data through the large model to determine an initial interaction intention; Decoding the video file to obtain multiple decoded video frames; Determining a key frame associated with the initial interaction intent from the plurality of decoded video frames; and The interaction output data is determined based on the key frame and the initial interaction intention.

5. The method according to claim 2, wherein: The adverse impact is associated with at least one of the following: a network condition on the large model side, a network condition on the interactive object side that interacts with the large model in the video interaction, and a condition of the interactive input data.

6. The method according to claim 5, wherein: In the case where the adverse effect is associated with the network condition of the large model side, the preset condition at least includes that the network transmission rate of the large model is lower than a first threshold; and wherein, In the case where the adverse impact is associated with the network status of the interactive object, the preset situation at least includes that the network transmission rate of the interactive object is lower than a second threshold.

7. The method according to claim 5, wherein: In response to determining that the adverse effect is associated with the status of the interactive input data, the preset situation includes at least one of the following: In the case where the interactive input data is image data or video data, the clarity of the interactive input data is lower than a third threshold, and In the case that the interactive input data is voice data or video data, the interactive input data lacks data frames.

8. The method according to any one of claims 1 to 7, further comprising: Output prompt information, where the prompt information is used to instruct to upload the target file.

9. The method according to claim 2, further comprising: In response to detecting the existence of interest information of the interactive object of the video interaction, acquiring the interest information; as well as The interest information, the interaction input data and the target document are processed by the large model to determine the interaction output data.

10. A video interaction device based on a large model, comprising: The first module is configured to obtain input interactive input data and a target file during a video interaction process based on a large model, wherein the interactive input data includes any one of voice data, image data and video data; A second module processes the interactive input data and the target file through the large model to determine interactive output data; and The third module is configured to output the interactive output data.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

13. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images