Multimedia stream processing method and device and related equipment

Abstract

The invention provides a multimedia stream processing method, which comprises the steps of outputting a configuration interface, and obtaining first configuration information in response to an operation of a user on the configuration interface, the first configuration information being used for configuring a recording task, a detection algorithm and a recording requirement; obtaining a multimedia stream under the recording task, the multimedia stream being a video stream obtained by performing video recording on the collected object or an audio stream obtained by performing voice recording; and detecting the multimedia stream according to a detection algorithm configured by a user to obtain a detection result, and when the detection result does not meet a recording requirement configured by the user, outputting a recording suggestion for the collected object. Thus, by feeding back suggestions, the efficiency of generating the recorded video / audio meeting the user expectation can be effectively improved, and the differentiated requirements of different users for different recording tasks, different detection algorithms and different recording requirements can be met. In addition, the invention further provides a corresponding processing device and related equipment.

Description

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 11, 2023, with application number 202311319379.8 and invention name “Multimedia Stream Processing Method, Device and Related Equipment”, all contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of multimedia technology, and in particular to a method, apparatus and related equipment for processing multimedia streams. Background Art

[0003] At present, video recording or audio recording has been widely used in many business scenarios. Taking digital human as an example, digital human refers to a digital human image close to human image created by digital technology. In the process of generating digital human, usually by recording the expressions and body movements of real human, a digital human who can express emotions and movements is generated, or by recording the voice of real human, a digital human who outputs human voice is generated.

[0004] At present, after completing video recording, users usually re-watch the recorded video, and when there are defects in the video, such as inadequate expression of body movements in the recorded video, the user will re-execute the process of recording the video. In this way, through repeated recording and repeated viewing, a video that meets the user's expectations is finally obtained. Similarly, the user can obtain audio that meets the user's expectations by repeatedly recording and listening to audio. However, this method of generating recorded videos or recorded audio that meets the user's expectations is inefficient and has high labor costs. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a method for processing multimedia streams to improve the accuracy of determining the root cause of an abnormality for an object. The present application also provides a corresponding data processing device, a computing device cluster, a computer-readable storage medium, and a computer program product.

[0006] In the first aspect, the embodiment of the present application provides a method for processing a multimedia stream, which can be executed by a corresponding processing device. Specifically, the processing device outputs a configuration interface, such as outputting a configuration interface to a client provided to the outside, and the configuration interface is used to prompt the configuration of recording tasks, detection algorithms, and recording requirements. Among them, the recording task, for example, can be a video recording task for changes in body movements, a video recording task for changes in facial expressions, a video recording task for action redirection, an audio recording task for the voice of a person or an animal, etc.; the detection algorithm, for example, can be an algorithm for detecting body movements expressed by multiple frames in a video stream; the recording requirement, for example, can be a requirement that the hand movement remains in a "still" state for a duration of not less than 2 seconds, etc. Then, the processing device responds to the user's operation on the configuration interface, and obtains the first configuration information provided by the user, and the first configuration information is used to configure the recording task, the detection algorithm, and the recording requirement. Next, the processing device obtains the multimedia stream under the recording task configured by the user, and the multimedia stream can be a video stream or an audio stream, the video stream is obtained by recording the video of the collected object, and the audio stream is obtained by recording the voice of the collected object. Finally, the processing device detects the multimedia stream according to the detection algorithm configured by the user, obtains the detection result, and determines whether the detection result meets the recording requirement configured by the user. When the detection result does not meet the recording requirement configured by the user, the processing device outputs a recording suggestion for the object being collected, such as suggesting that the object being collected re-record part of the content.

[0007] Since the processing device can automatically process the recorded video stream / audio stream and output recording suggestions according to the recording requirements, the user does not need to determine the problem in the video / audio by repeatedly watching the video or listening to the audio, but can directly re-record the video or audio based on the output recording suggestions, thereby effectively improving the efficiency of generating recorded video / audio that meets the user's expectations. In actual application, during the process of recording video or audio, the recording suggestions for the current recording stage can be output in real time according to the recording requirements, so that the user can re-record only the current recording stage without performing the re-recording process of all recording stages, which can further improve the efficiency of generating recorded video / audio. Moreover, the processing device can support the user to configure the recording task, detection algorithm and recording requirements independently, which can meet the differentiated needs of different users for different recording tasks, different detection algorithms and different recording requirements, thereby improving the flexibility of users in generating recorded video / audio and enriching the application scenarios applicable to audio and video recording.

[0008] In a possible implementation, the recording requirements configured by the user are used to constrain the requirements that the video content in the recorded video stream meets, or to constrain the requirements that the audio content in the recorded audio stream meets. In this way, the user can customize the requirements for video recording or audio recording to meet the needs of the user in different business scenarios (that is, for the same type of recording task, the user may have different requirements for the recorded video or audio in different scenarios).

[0009] In one possible implementation, the recording task configured by the user includes a limb motion capture task. In this case, the multimedia stream is a video stream, which is obtained by recording the limb motions (such as hand motions) of the captured object. The detection algorithm configured by the user is used to detect the limb motions in the video stream, and the recording requirements configured by the user are used to constrain the requirements to be met by the limb motions in the video stream.

[0010] In one possible implementation, the recording task configured by the user includes an expression capture task, the multimedia stream is a video stream, the video stream is obtained by recording the facial expressions of the captured object, the detection algorithm configured by the user is used to detect the facial expressions in the video stream, and the recording requirements configured by the user are used to constrain the requirements met by the facial expressions in the video stream.

[0011] In one possible implementation, the recording task configured by the user includes a motion redirection task, the multimedia stream is a video stream, and the video stream is obtained by recording the body movements of the captured object. The detection algorithm configured by the user is used to detect motion problems existing in redirecting the body movements in the video stream to the virtual character, and the recording requirements configured by the user are used to constrain the requirements met by the body movements redirected to the virtual character in the video stream.

[0012] In one possible implementation, the recording task configured by the user includes a voice collection task, the multimedia stream is an audio stream, and the audio stream is obtained by recording the pronunciation of the collected object. The detection algorithm configured by the user is used to detect the voice meaning (such as pronunciation style, emotion, etc.) in the audio stream, and the recording requirements configured by the user are used to constrain the requirements met by the voice meaning in the audio stream.

[0013] In a possible implementation, the configuration interface output by the processing device is also used to prompt the user to configure the form of expression of the recording suggestions. Then, the client provided by the processing device to the outside can respond to the user's operation on the configuration interface and obtain the second configuration information for the form of expression of the recording suggestions provided by the user; in this way, when the processing device outputs the recording suggestions for the collected object, it can be that when the detection result does not meet the recording requirements, it will first generate the recording suggestions in the first form; then, the processing device transforms the recording suggestions in the first form according to the second configuration information configured by the user, obtains the recording suggestions in the second form, and outputs the recording suggestions in the second form. In this way, not only can the diversity of the recording suggestions output by the processing device be enriched, but also the user's selectivity for the form of expression of the recording suggestions can be increased, thereby further improving the user experience.

[0014] In a possible implementation, when the processing device obtains the second configuration information for the expression form of the recording suggestion provided by the user, it may specifically obtain the second configuration file input by the user on the configuration interface, and the second configuration file includes the second configuration information; or, the client provided by the processing device to the outside may obtain the second configuration information configured by the user in response to the user's selection operation for the identification of multiple candidate expression forms. In this way, the expression form of the recording suggestion can be configured by the user, and the user's selectability for the expression form of the recording suggestion can be improved, thereby further improving the user experience.

[0015] In a possible implementation, when the processing device obtains the first configuration information provided by the user, it can specifically determine the recording task configured by the user in response to the user's selection operation on the configuration interface for multiple candidate recording task identifiers; and determine the detection algorithm configured by the user in response to the user's selection operation on the configuration interface for multiple candidate detection algorithm identifiers; and determine the recording requirement configured by the user in response to the user's selection operation on the configuration interface for multiple candidate recording requirement identifiers. In this way, the user can flexibly select the recording task, detection algorithm and recording requirement on the configuration interface, thereby improving the flexibility of information configuration and user experience.

[0016] In a possible implementation, multiple candidate detection algorithms and multiple candidate recording requirements for user selection are stored in a database, and the processing device can also output an update interface through the client, and the update interface is used to prompt the database to be updated, so that the processing device can respond to the user's update operation on the update interface for the database, and update the candidate detection algorithms or candidate recording requirements in the database. In this way, through dynamic data updates, the candidate recording requirements and candidate detection algorithms in the database can adapt to changes in the user's demand for detection algorithms and recording requirements, thereby further improving the user experience.

[0017] In a possible implementation, when the processing device obtains the first configuration information provided by the user, it may specifically obtain a first configuration file input by the user on the configuration interface, and the first configuration file includes a recording task, a detection algorithm, or a recording requirement. In this way, the user can configure the recording task, the detection algorithm, or the recording requirement by importing the configuration file, thereby enabling the user to customize the configuration, improving the flexibility of the user configuration and the user experience.

[0018] In a possible implementation, the multimedia stream acquired by the processing device includes a video stream, and the processing device may further fuse the detection result or recording suggestion with the video stream to obtain a fused video stream, and output the fused video stream. In this way, the user can view the detection status of the processing device for the content in the video stream, or the user can view the video portion in the video stream that needs to be re-recorded, so as to subsequently guide the captured object to re-record the portion of the video, thereby improving the user experience.

[0019] In a second aspect, the present application provides a multimedia stream processing device, which includes: an acquisition module, which is used to output a configuration interface, the configuration interface is used to prompt the configuration of a recording task, a detection algorithm, and a recording requirement, and in response to the user's operation on the configuration interface, obtain first configuration information provided by the user, the first configuration information is used to configure the recording task, the detection algorithm, and the recording requirements, and obtain the multimedia stream under the recording task, the multimedia stream includes a video stream or an audio stream, the video stream is obtained by recording the video of the collected object, and the audio stream is obtained by recording the voice of the collected object; a detection module, which is used to detect the multimedia stream according to the detection algorithm to obtain a detection result; an output module, which is used to output a recording suggestion for the collected object when the detection result does not meet the recording requirements.

[0020] In a possible implementation manner, the recording requirement is used to constrain requirements that video content in a recorded video stream must meet, or is used to constrain requirements that audio content in a recorded audio stream must meet.

[0021] In a possible implementation, the recording task includes a body movement collection task, the multimedia stream is a video stream, the detection algorithm is used to detect body movements in the video stream, and the recording requirements are used to constrain the requirements met by the body movements in the video stream.

[0022] In a possible implementation, the recording task includes an expression collection task, the multimedia stream is a video stream, the detection algorithm is used to detect facial expressions in the video stream, and the recording requirements are used to constrain requirements satisfied by facial expressions in the video stream.

[0023] In one possible implementation, the recording task includes an action redirection task, the multimedia stream is a video stream, the detection algorithm is used to detect action problems existing in redirecting body movements in the video stream to a virtual character, and the recording requirements are used to constrain the requirements met by body movements redirected to the virtual character in the video stream.

[0024] In a possible implementation, the recording task includes a voice collection task, the multimedia stream is an audio stream, the detection algorithm is used to detect the voice meaning in the audio stream, and the recording requirement is used to constrain the requirements met by the voice meaning in the audio stream.

[0025] In one possible implementation, the configuration interface is also used to prompt the configuration of the expression form of the recording suggestions; the acquisition module is also used to obtain second configuration information for the expression form of the recording suggestions provided by the user in response to the user's operation on the configuration interface; the output module is used to: when the detection result does not meet the recording requirements, generate a recording suggestion in a first expression form; according to the second configuration information, transform the recording suggestion in the first expression form to obtain a recording suggestion in a second expression form; and output the recording suggestion in the second expression form.

[0026] In a possible implementation, the acquisition module is used to: acquire a second configuration file input by a user on a configuration interface, the second configuration file including second configuration information; or, in response to a user's selection operation for identification of multiple candidate representation forms, obtain the second configuration information configured by the user.

[0027] In one possible implementation, the acquisition module is used to: determine the recording task configured by the user in response to the user's selection operation on the configuration interface for multiple candidate recording task identifiers; determine the detection algorithm configured by the user in response to the user's selection operation on the configuration interface for multiple candidate detection algorithm identifiers; determine the recording requirement configured by the user in response to the user's selection operation on the configuration interface for multiple candidate recording requirement identifiers.

[0028] In one possible implementation, multiple candidate detection algorithms and multiple candidate recording requirements are stored in a database, and the acquisition module is further used to: output an update interface, where the update interface is used to prompt the update of the database; and update the candidate detection algorithms or candidate recording requirements in the database in response to the user's update operation on the database on the update interface.

[0029] In a possible implementation, the acquisition module is used to acquire a first configuration file input by a user on a configuration interface, where the first configuration file includes a recording task, a detection algorithm, or a recording requirement.

[0030] In a possible implementation, the multimedia stream includes a video stream, and the output module is further used to: merge the detection result or recording suggestion with the video stream to obtain a merged video stream; and output the merged video stream.

[0031] The multimedia stream processing device provided in the second aspect corresponds to the multimedia stream processing method provided in the first aspect. Therefore, the technical effects of the multimedia stream processing device provided in the second aspect can be referred to the relevant description of the technical effects of the above-mentioned first aspect or any implementation method of the first aspect, and will not be repeated here.

[0032] In a third aspect, the present application provides a computing device cluster, the computing device cluster includes at least one computing device, the at least one computing device includes at least one processor and at least one memory; the at least one memory is used to store instructions, and the at least one processor executes the instructions stored in the at least one memory, so that the computing device cluster executes the multimedia stream processing method in the above-mentioned first aspect or any possible implementation of the first aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The at least one computing device may also include a bus. The processor is connected to the memory via a bus. The memory may include a readable memory and a random access memory.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on at least one computing device, the at least one computing device executes the multimedia stream processing method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0034] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on at least one computing device, enables the at least one computing device to execute the multimedia stream processing method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0035] Based on the implementations provided in the above aspects, the present application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0037] Figure 1 A schematic diagram of the structure of an exemplary multimedia stream processing system provided by the present application;

[0038] Figure 2 A flowchart of a multimedia stream processing method provided by the present application;

[0039] Figure 3 A schematic diagram of an exemplary configuration interface provided by this application;

[0040] Figure 4 A schematic diagram of a video stream processing method provided in this application;

[0041] Figure 5 A flowchart of another video stream processing method provided by this application;

[0042] Figure 6 A flowchart of another video stream processing method provided by the present application;

[0043] Figure 7 A flowchart of an audio stream processing method provided by the present application;

[0044] Figure 8 A schematic diagram of the structure of a computing device provided for this application;

[0045] Fig. 9 A schematic diagram of the structure of a computing device cluster provided in this application. DETAILED DESCRIPTION

[0046] The scheme in the embodiments provided in this application will be described below in conjunction with the drawings in this application.

[0047] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.

[0048] See also Figure 1 , is a schematic diagram of the structure of an exemplary multimedia stream processing system 10. Figure 1 As shown, the multimedia stream processing system 10 includes a collection device 100, a processing device 200 and a client 300, and the collection device 100, the processing device 200 and the client 300 can communicate with each other through a network. The client 300 can be an application running on a user-side device, or can be a web browser provided by the multimedia stream processing system 10.

[0049] The acquisition device 100 is used to acquire multimedia streams, which may specifically be video streams or audio streams. For example, the acquisition device 100 may be an infrared camera, a depth camera, etc., so that by recording the captured object (such as a person), a corresponding video stream can be obtained; or the acquisition device 100 may be a microphone, etc., so that by recording the captured object, a corresponding audio stream can be obtained. Whether to record or record the captured object can be determined based on the recording task configured by the user through the client 300.

[0050] In actual application, the collection device 100 can be deployed in an integrated manner with the client 300. For example, the collection device 100 can be a terminal device with a shooting or recording function, and the client 300 can be deployed on the terminal device as an application; or, the collection device 100 and the client 300 can also be deployed separately. In this case, the collection device 100 can send the collected multimedia stream to the processing device 200 through the client 300, or the collection device 10 can also directly send the multimedia stream to the processing device 200.

[0051] The processing device 200 is used to process the received multimedia stream and output corresponding recording suggestions, which can be used to guide the captured object to perform audio or video recording, such as guiding the captured object to follow the recording requirements to complete the video recording of facial expressions.

[0052] Specifically, Figure 1As shown, the processing device 200 may include an acquisition module 201, a detection module 202, and an output module 203. Among them, the acquisition module 201 can receive a configuration interface output by the client 300, and the configuration interface is used to prompt the user to configure the recording task, the detection algorithm, and the recording requirements, so that the client 300 responds to the user's operation on the configuration interface, obtains the configuration information provided by the user for the recording task, the detection algorithm, and the recording requirements, and also receives the multimedia stream under the recording task sent by the client 300, and provides the multimedia stream to the detection module 202, and provides the recording requirements to the output module 203. The detection module 202 detects the multimedia stream according to the detection algorithm to obtain the detection result. For example, the multimedia stream is a video stream for recording facial expressions, and the detection result is specifically the detection results of "open mouth", "closed mouth", "open eyes", "closed eyes", etc. recognized by the processing device 200. Then, the detection module 202 provides the detection result to the output module 203. The output module 203 determines whether the detection result meets the recording requirements, and when the detection result does not meet the recording requirements, a recording suggestion for the collected object is generated. For example, the output module 203 may detect that the facial expression in the multimedia stream does not have "closed eyes" (i.e., does not meet the recording requirements), then a recording suggestion for instructing the collected object to make an "open eyes" expression can be generated. Finally, the output module 203 can output the recording suggestion to the client 300, so that the client 300 presents the recording suggestion to the user or the collected object.

[0053] Since the processing device 200 can automatically process the recorded video stream / audio stream and output recording suggestions according to the recording requirements, the user does not need to determine the problems in the video / audio by repeatedly watching the video or repeatedly listening to the audio, but can directly re-record the video or audio based on the output recording suggestions, thereby effectively improving the efficiency of generating recorded video / audio that meets the user's expectations.

[0054] In actual application, during the process of recording video or audio, the acquisition device 100 can (through the client 300) continuously provide multimedia streams to the processing device 200. In this way, the processing device 200 can output recording suggestions for the current recording stage in real time according to the recording requirements, which allows the user to re-record only the current recording stage without having to perform the re-recording process of all recording stages, thereby further improving the efficiency of generating recorded video / audio.

[0055] Moreover, the processing device 200 can support users to independently configure recording tasks, detection algorithms and recording requirements, that is, users can independently select from a variety of candidate items corresponding to recording tasks, detection algorithms and recording requirements according to actual application needs, or users can customize recording tasks, detection algorithms and recording requirements, which can meet the differentiated needs of different users for different recording tasks, different detection algorithms and different recording requirements, thereby improving the flexibility of users in generating recorded videos / audio and enriching the application scenarios applicable to audio and video recording.

[0056] It is worth noting that Figure 1 The multimedia stream processing system 10 shown is only used as an exemplary description and is not intended to limit the specific implementation of the multimedia stream processing system. Figure 1 In the multimedia stream processing system 10 shown, the processing device 200 can be deployed in the cloud, such as in a public cloud, a hybrid cloud, etc., to provide users with cloud services for processing multimedia streams, and the collection device 100 and the client can be deployed on the user side to facilitate the user to provide corresponding configuration information locally and generate corresponding multimedia streams. In other possible multimedia stream processing systems, only the collection device 100 and the processing device 200 may be included, and the collection device 100 and the processing device 200 are both deployed on the user side, so that the processing device 200 can provide users with localized services for processing multimedia streams; at the same time, the processing device 200 can present an interactive interface to the user, so that the user can perform human-computer interaction with the processing device 200 through the interactive interface. Alternatively, in other possible multimedia stream processing systems, the multimedia stream processing system may include multiple collection devices and provide multiple clients to the outside to realize parallel processing of multiple multimedia streams. Alternatively, the multimedia stream processing system may include other types of devices to support the multimedia stream processing system to have more functions. The present application does not limit the specific architecture of the multimedia stream processing system.

[0057] In addition, the processing device 200 may be implemented by software or hardware. As an example, the implementation of the processing device 200 is described below.

[0058] As an example of a software functional unit, the processing device 200 may include code running on an instance, such as an agent running on an instance, and the instance may be at least one of a host, a virtual machine, a container, a thread, and a process. Furthermore, the above instance may be one or more. For example, the processing device 200 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Among them, generally a region may include multiple AZs.

[0059] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0060] The processing device 200 is taken as an example of a hardware functional unit, and the processing device 200 may include at least one computing device, such as a cloud server, etc. Alternatively, the processing device 200 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The above-mentioned PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU) or any combination thereof.

[0061] When the processing device 200 may include multiple computing devices, the multiple computing devices may be distributed in the same region or in different regions. The multiple computing devices included in the processing device 200 may be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the processing device 200 may be distributed in the same VPC or in multiple VPCs. The multiple computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0062] It should be noted that the method steps executed by each module in the above-mentioned processing device 200 are only used as an exemplary description and are not intended to be limiting. For example, in other implementation examples, the acquisition module 201 in the processing device 200 can be used to execute the method steps executed by any module in the above-mentioned processing device 200, the detection module 202 can be used to execute the method steps executed by any module in the above-mentioned processing device 200, and the output module 203 can be used to execute the method steps executed by any module in the above-mentioned processing device 200. The steps that the acquisition module 201, the detection module 202 and the output module 203 are responsible for implementing can be specified as needed, and the full functions of the processing device 200 are realized by respectively implementing different method steps in the process of processing the multimedia stream by the acquisition module 201, the detection module 202 and the output module 203.

[0063] To facilitate understanding, various non-limiting specific implementations of the multimedia stream processing process are described in detail below.

[0064] See also Figure 2 , is a flowchart of a method for processing a multimedia stream in an embodiment of the present application. The method can be applied to the above Figure 1 The multimedia stream processing system 10 shown in FIG. 1 may also be applied to other applicable multimedia stream processing systems. Figure 1 Taking the multimedia stream processing system 10 as an example, the process of the modules in the processing device 200 processing the multimedia stream and outputting the recording suggestion is introduced.

[0065] like Figure 2 As shown, the method for processing the multimedia stream may specifically include:

[0066] S201: The acquisition module 201 outputs a configuration interface to the client 300, where the configuration interface is used to prompt the configuration of the recording task, the detection algorithm and the recording requirements.

[0067] S202: The client 300 sends the configuration information provided by the user to the acquisition module 201 in response to the user's operation on the configuration interface. The configuration information is used to configure the recording task, the detection algorithm and the recording requirements.

[0068] In this embodiment, the multimedia stream processing system 10 can support users to perform various recording tasks, such as video recording tasks for changes in body movements, video recording tasks for changes in facial expressions, video recording tasks for action redirection (redirecting real human body movements to characters in the virtual world), audio recording tasks for human or animal voices, etc., so that users can select corresponding tasks from the various recording tasks according to the needs of actual applications.

[0069] Exemplarily, the acquisition module 201 may output to the client 300: Figure 3 The configuration interface shown in the figure prompts the user to configure the recording task on the configuration interface. Figure 3 As shown, the configuration interface may include identifiers of multiple candidate recording tasks supported by the multimedia stream processing system 10, and the candidate recording tasks may be, for example, a body movement collection task, an expression collection task, an action redirection task, a voice collection task, etc. In this way, the user can select the identifiers of the multiple candidate recording tasks on the configuration interface, so that the client 300 can provide the identifier of the candidate recording task selected by the user to the acquisition module 201 in response to the user's selection operation, so that the acquisition module 201 determines the recording task configured by the user, that is, the candidate recording task indicated by the identifier selected by the user, according to the identifier.

[0070] Furthermore, when the multiple candidate recording tasks presented on the configuration interface do not meet the user's recording needs, such as the video content that the user needs to record is the running action of an animal (which does not match the above recording task), the user can also customize the recording task on the configuration interface. Figure 3 The configuration file is input on the configuration interface shown, and the configuration file includes a customized recording task, etc. Exemplarily, the configuration file can be a json format file, or a configuration file in another format. Then, the client 300 can send the configuration file provided by the user to the acquisition module 201, so that the acquisition module 201 determines the recording task configured by the user according to the configuration file.

[0071] In addition to configuring recording tasks, users can also configure detection algorithms and recording requirements on the configuration interface presented by the client.

[0072] Among them, the detection algorithm is used to detect the recorded video stream or audio stream, such as detecting the body movements expressed by multiple frames in the video stream, or detecting the environmental noise, pronunciation emotions, etc. in the voice expressed in the audio stream.

[0073] Recording requirements are used to constrain the requirements that the content of the recorded video stream or audio stream must meet. For example, for recording a video of hand movements, the recording requirement may specifically require that the hand movement remain in a "still" state for at least 2 seconds.

[0074] Exemplarily, after the user completes the configuration of the recording task, the acquisition module 201 may further output to the client 300 the identifiers of one or more candidate detection algorithms and one or more candidate recording requirements corresponding to the recording task according to the recording task configured by the user, and prompt the user to configure the detection algorithm and the recording requirements. Figure 3 As shown, the client 300 can also present the identifiers of multiple candidate detection algorithms and multiple recording requirements supported by the multimedia stream processing system 10 on the configuration interface. In this way, the user can continue to select the identifiers of multiple candidate detection algorithms and multiple candidate recording requirements on the configuration interface. Accordingly, the client 300 can provide the identifier of the detection algorithm task selected by the user and the identifier of the candidate recording task to the acquisition module 201 in response to the user's selection operation, so that the acquisition module 201 determines the detection algorithm configured by the user (that is, the candidate detection algorithm indicated by the identifier selected by the user) and determines the recording requirement configured by the user (that is, the candidate recording requirement indicated by the identifier selected by the user) based on these identifiers.

[0075] Furthermore, when the multiple candidate detection algorithm identifiers and candidate recording requirement identifiers presented on the configuration interface do not meet the user's needs, the user can also customize the detection algorithm and recording requirement on the configuration interface. Figure 3 The configuration file inputted on the configuration interface shown includes not only the name of the customized recording task, but also the user-defined detection algorithm and the user-defined recording requirements. In this way, the acquisition module 201 can obtain the user-configured detection algorithm and recording requirements from the configuration file sent by the client 300.

[0076] It should be noted that the above implementation method of obtaining configuration information by the obtaining module 201 is only for exemplary description and is not intended to be limiting. In actual application, the obtaining module 201 may also obtain configuration information in other ways.

[0077] After receiving the configuration information, the acquisition module 201 may provide the identification of the detection algorithm configured by the user to the detection module 202 , and provide the identification of the recording requirement configured by the user to the output module 203 .

[0078] S202: The acquisition module 201 acquires the multimedia stream under the recording task configured by the user, and the multimedia stream includes a video stream or an audio stream, wherein the video stream is obtained by recording the video of the captured object, and the audio stream is obtained by recording the voice of the captured object.

[0079] After the user completes the configuration of the recording task, the acquisition module 201 may further acquire the video stream or audio stream of the recording task. For the convenience of description, the acquired video stream or audio stream is collectively referred to as multimedia stream below.

[0080] In a possible implementation, after receiving the configuration information sent by the client 300, the acquisition module 201 may send a request to the client 300 to acquire a multimedia stream. The client 300 responds to the request and uses the acquisition device 100 to record the video or audio of the object to be acquired under the recording task configured by the user to obtain the multimedia stream under the recording task. The object to be acquired refers to the object whose video or audio is recorded by the acquisition device 100, which may be a person or an animal, or other types of objects.

[0081] In actual application, during the process of recording the collected object, the client 300 can present an interactive interface to guide the collected object to record video or audio. For example, when the recording task is specifically a body movement collection task, the client 300 can present prompt information of the body movement (such as the name of the body movement, etc.) on the interactive interface. In this way, the collected object can make corresponding body movements according to the prompt information; and, during the process of the collected object making body movements, the collection device 100 can continue to shoot the collected object to record the behavior video of the collected object making body movements. For another example, when the recording task is specifically a voice collection task, the client 300 can present a text on the interactive interface, and the collected object can pronounce according to the text content presented on the interactive interface; in this way, the collection device 100 can continue to collect the voice of the collected object to achieve audio recording for the collected object.

[0082] Then, the acquisition device 100 can send the acquired multimedia stream to the client 300, so that the client 300 can transmit the multimedia stream to the acquisition module 201 in the processing device 200, so that the acquisition module 201 obtains the multimedia stream. In other possible implementations, the acquisition device 100 has wireless transmission capability, so that the acquisition device 100 can also remotely transmit the multimedia stream to the processing device 200 through the network.

[0083] In the process of recording the video or audio of the collected object, the collection device 100 continuously sends the multimedia stream to the processing device 200 (or to the processing device 200 through the client 300). Alternatively, after the collection device 100 completes the video or audio collection of the collected object within a period of time, the collected audio or video can be sent to the processing device 200 (or to the processing device 200 through the client 300).

[0084] After acquiring the multimedia stream, the acquisition module 201 may provide the multimedia stream to the detection module 202 .

[0085] S203: The detection module 202 detects the multimedia stream according to the detection algorithm configured by the user to obtain a detection result.

[0086] In a possible implementation, when the configuration information acquired by the acquisition module 201 includes an identifier of a detection algorithm, the detection module 202 can access a database according to the identifier of the detection algorithm, and the database pre-stores a plurality of detection algorithms (i.e., the candidate detection algorithms described above), so that the detection module 202 can access the detection algorithm configured by the user from the database. In addition, the database can also store a detection configuration corresponding to each detection algorithm, and the detection configuration can be, for example, a value range of a plurality of meaning types, and the value range of each meaning type can be used to constrain the value of the detection result of the same meaning. For example, when the moving distance of the hand position of the detection algorithm is within the value range of meaning type 1, it indicates that the hand position has not moved; and when the moving distance of the hand position of the detection algorithm is within the value range of meaning type 2, it indicates that the hand position has moved. Thus, the detection module 202 can detect the multimedia stream using the accessed detection algorithm and the detection configuration corresponding to the detection algorithm.

[0087] When the user configures the detection algorithm by uploading a configuration file, the detection module 202 can detect the multimedia stream according to the detection configuration in the configuration file by running the user-defined detection algorithm in the configuration file.

[0088] When detecting the multimedia stream, the detection module 202 may specifically detect the meaning expressed by the multimedia stream. Further, the detection module 202 may also detect attribute information related to the meaning. This is exemplified below with reference to an example.

[0089] In the first example, the multimedia stream can be a video stream obtained by recording the body movements of the collected object. At this time, the detection algorithm configured by the user is specifically an algorithm for detecting body movements in the video stream, and the body movements can be, for example, "clenching fists" and "clapping" of the hands, or can be "lowering the head", "raising the head", "turning the head" and other movements of the head. Then, in the process of detecting the video stream using the detection algorithm, the detection module 202 can first identify the body in each frame of the video stream, and identify the body movements expressed by the multiple frames according to the differences between the body images in the continuous multiple frames; and the detection module 202 can also calculate the relevant attribute information of the body movement according to the multiple frames, such as the time taken by the collected object to make the body movement (which can be calculated by the number of the multiple frames), the angle and amplitude of the body movement, and other information. Thus, the detection module 202 can generate a detection result including each body movement and the relevant attributes of each body movement.

[0090] In the second example, the multimedia stream may specifically be a video stream obtained by recording the facial expressions of the collected object. At this time, the detection algorithm configured by the user is specifically an algorithm for detecting facial expressions in the video stream, and the facial expressions may be, for example, expressions such as "open mouth", "closed mouth", "open eyes", "closed eyes", "frown", etc. Then, in the process of detecting the video stream using the detection algorithm, the detection module 202 can identify the facial expressions included in each frame of the video stream; and the detection module 202 can also detect the relevant attribute information of the facial expressions in each frame, such as the angle, amplitude, etc. of the facial expressions. Thus, the detection module 202 can generate a detection result including the facial expressions corresponding to multiple frames and the relevant attributes of each facial expression.

[0091] In the third example, the multimedia stream may be a video stream obtained by recording the body movements of the collected object. At this time, the detection algorithm configured by the user is specifically an algorithm for detecting the problem of redirecting body movements to the action of the virtual character, and the body movements may be, for example, "walking", "running", "bowing", "holding fists" and other actions. Then, in the process of detecting the video stream using the detection algorithm, the detection module 202 may first redirect the body movements of the collected object in the video stream to the body movements of the virtual character (such as digital people, etc.), so as to generate a video stream of the virtual character; and then identify the body in each frame of the video stream of the virtual character, and check whether there is a problem of penetration in each frame (such as mutual penetration and superposition of body A and body B, mutual penetration and superposition of body and clothes of the virtual character, etc.), and check whether there is an abnormality in the body movements expressed by multiple consecutive frames (such as a sudden change between the body movements in the previous frame and the next frame, etc.). Furthermore, the detection module 202 can further detect the duration of time when the video stream of the virtual character does not have abnormalities, or detect the duration of time when the video stream has model penetration problems, the duration of time when the body movements are abnormal, the cause of the abnormality, and other related attribute information based on the multiple frames. Thus, the detection module 202 can generate a detection result including the video stream of the virtual character and the related attributes of the video stream.

[0092] In the fourth example, the multimedia stream can specifically be an audio stream obtained by audio recording the voice of the collected object. At this time, the detection algorithm configured by the user is specifically an algorithm for detecting the voice meaning of the collected object, and the voice meaning can be, for example, environmental noise, echo, phoneme, voice emotion, voice style, etc. present in the voice. Then, the detection module 202 can use the detection algorithm to detect voice meaning information such as environmental noise, echo, phoneme, voice emotion, or voice style in the audio stream. Further, the detection module 202 can also detect attribute information related to the voice meaning information, such as further detecting the decibel and duration of the environmental noise, detecting the number of types of voice emotions, and other related attribute information. Thus, the detection module 202 can generate a detection result including voice meaning and related attributes of voice meaning.

[0093] It is understood that the above four implementations are only exemplary and not limiting. For example, in other possible implementations, the detection module 202 may not need to detect the relevant attributes of the meaning expressed by the multimedia stream, or the meaning expressed by the multimedia stream detected by the detection module 202 may also be other types of meanings.

[0094] After generating the detection result, the detection module 202 may provide the detection result to the output module 203 .

[0095] S204: The output module 203 determines whether the detection result meets the recording requirement.

[0096] S205: When the detection result does not meet the recording requirement, the output module 203 outputs a recording suggestion for the captured object to the client 300.

[0097] In actual application scenarios, the content of the multimedia stream obtained by recording the video or audio of the captured object is likely to not meet the user's expectations. For example, when recording the body movements of the captured object, the body movements in the recorded video stream may not contain the "head turning" movement expected by the user, or the "head turning" movement exists in the video stream but the turning angle of the "head turning" movement may not reach the 60 degrees expected by the user.

[0098] Therefore, after obtaining the detection result, the output module 203 can further verify whether the detection result meets the recording requirements configured by the user. For example, the output module 203 can verify whether there is a "head turning" action in the video stream based on the detection result, and when it is determined that there is a "head turning" action, the output module 203 can further verify whether the head rotation angle corresponding to the "head turning" action in the video stream reaches 60 degrees expected by the user.

[0099] When it is determined that the detection result does not meet the recording requirements, the output module 203 can generate corresponding recording suggestions based on the detection result and the recording requirements. For example, when the head rotation angle corresponding to the "turn head" action in the video stream does not reach 60 degrees indicated by the recording requirements, the recording suggestion generated by the output module 203 can be "rotate the head to the left (or right)", or it can be "continue to rotate the head to the left (or right) until it reaches 60 degrees". For another example, when it is detected that the voice emotion of part of the audio in the audio stream does not meet the emotional requirements (i.e., recording requirements) configured by the user, the recording suggestion generated by the output module 203 can be "re-record the voice of the XXXX text".

[0100] Finally, the output module 203 can output the recording suggestion to the client 300, such as Figure 2 As shown, the client 300 presents the recording suggestion to the user. In this way, the user can guide the subject to re-record the video or audio according to the presented recording suggestion, or re-record the part of the video clip or the part of the audio clip with problems. Alternatively, the user can perceive the problems in the multimedia stream in time according to the recording suggestion, so that the user can decide whether to use the multimedia stream to generate a digital human.

[0101] In actual application, the output module 203 can also output the recording suggestion to the collection device 100, such as Figure 2As shown, the collection device 100 presents the recording suggestion to the collected object, such as outputting the recording suggestion content in audio form to the collected object, so that the collected object can re-execute the video or audio recording process according to the recording suggestion.

[0102] In a further possible implementation, the output module 203 may also transform the presentation form of the generated recording suggestion, so that the recording suggestion output by the output module 203 can ultimately be presented to the user in a more user-friendly presentation form, thereby improving the user experience.

[0103] In specific implementation, the output module 203 generates recording suggestions in the first form by default according to the detection results and the recording requirements, such as generating recording suggestions in text form by default, or generating recording suggestions in numerical form by default, etc. Then, the output module 203 can transform the recording suggestions in the first form to obtain recording suggestions in the second form, and the second form can be, for example, a recording suggestion in the form of a picture, or a recording suggestion in the form of a voice, etc. Then, the output module 203 outputs the recording suggestions in the second form, such as outputting the recording suggestions in the second form to the client 300 or the collection device 100, etc. Among them, the second form can be any form that can improve the user experience, and can be set according to the needs of the actual application, and this specific implementation is not limited.

[0104] For example, the second form of the recording suggestion can be configured by the user. For example, before the output module 203 outputs the recording suggestion, the acquisition module 201 can output the following to the client 300: Figure 3 The configuration interface shown in the figure can also prompt the user to configure the expression form of the recording suggestion, so that the user can perform the configuration operation for the expression form of the recording suggestion on the client 300. For example, the user can select the identifiers of multiple candidate expression forms on the configuration interface, so that the acquisition module 201 can determine the second expression form configured by the user according to the identifier selected by the user; or, the user can enter a second configuration file on the configuration interface, and the second configuration file includes second configuration information for configuring the expression form. In this way, the acquisition module 201 can provide the second configuration information provided by the user to the output module 203, so that the output module 203 can transform the recording suggestion of the first expression form into the recording suggestion of the second expression form configured by the user. In this way, not only can the diversity of the recording suggestions output by the processing device 200 be enriched, but also the user's selectivity for the expression form of the recording suggestions can be improved, thereby further improving the user experience.

[0105] In other possible implementation examples, the second expression form of the recording suggestion may also be pre-configured in the processing device 200 by a technician, so that the output module 203 can output the recording suggestion in a fixed expression form, which is not limited in this embodiment.

[0106] Furthermore, in addition to outputting recording suggestions, the output module 203 can also output processed multimedia streams, such as Figure 2 As shown. For example, the acquisition module 201 can provide the multimedia stream sent by the client 300 to the output module 203, so that the output module 203 can merge the recording suggestion or the detection result generated by the detection module 202 with the multimedia stream, and output the merged multimedia stream to the client 300. Taking the multimedia stream as a video stream as an example, the output module 203 can add the detection result to the multi-frame picture in the video stream, such as marking the body movement / facial expression / position of the penetration detected by the detection module 202 in the multi-frame picture included in the video stream, and then the output module 203 outputs a new video stream fused with the detection result. In this way, the user can view the detection situation of the processing device 200 for the content in the video stream. Alternatively, the output module 203 can add the recording suggestion to the multi-frame picture in the video stream, such as adding the recording suggestion to the multi-frame picture of the suggestion to re-record included in the video stream. In this way, the user can view the video part that needs to be re-recorded in the video stream, so as to guide the collected object to re-record the part of the video later.

[0107] In addition, the user can also update the candidate detection algorithms and candidate recording requirements stored in the database. In specific implementation, the acquisition module 201 can also output an update interface to the client 300 so that the client 300 presents the update interface to the user and prompts the user to update the database storing the candidate detection algorithms and candidate recording requirements. In this way, the user can perform update operations on the database on the update interface, such as providing a new detection algorithm or a new recording requirement to the processing device 200 through the client 300, or adjusting the existing candidate detection algorithm or candidate recording requirement. Accordingly, the acquisition module 201 can update the candidate detection algorithms and candidate recording requirements in the database in response to the selection operation performed by the user on the update interface, such as adding, deleting or modifying the candidate detection algorithms and candidate recording requirements in the database.

[0108] Above Figure 2The method steps and the execution order between the steps shown are only used as an implementation example and are not intended to be limiting. For example, in other possible embodiments, step S201 can be split into multiple method steps, such as the user can complete the configuration of the recording task, the detection algorithm, and the recording requirements through different configuration interfaces. In this case, the step of obtaining the detection algorithm and the recording requirements by the acquisition module 201 can be located after the step of obtaining the multimedia stream by the acquisition module 201.

[0109] Since the processing device 200 can automatically process the recorded video stream / audio stream and output recording suggestions according to the recording requirements, the user does not need to determine the problem in the video / audio by repeatedly watching the video or repeatedly listening to the audio, but can directly re-record the video or audio based on the output recording suggestions, thereby effectively improving the efficiency of generating recorded video / audio that meets the user's expectations. In addition, the processing device 200 can support the user to independently configure the recording task, detection algorithm and recording requirements, that is, the user can independently select from a variety of candidate items corresponding to the recording task, detection algorithm and recording requirements (and forms of expression) according to the needs of the actual application, or the user can customize the recording task, detection algorithm and recording requirements (and forms of expression), which can meet the differentiated needs of different users for different recording tasks, different detection algorithms, and different recording requirements (and different forms of expression), thereby improving the flexibility of users in generating recorded video / audio and enriching the application scenarios applicable to audio and video recording.

[0110] To facilitate further understanding of the technical solution of the embodiment of the present application, the process of processing multimedia streams is introduced below in conjunction with a specific recording task.

[0111] See also Figure 4 , shows a schematic flow chart of a method for processing a video stream. Figure 4 In the method embodiment shown, the recording task may specifically be a task of recording a video of a body movement of the subject being captured, where the body movement is a hand movement, such as a "fisting" movement or a "hand-free" movement. Figure 4 As shown, the method may specifically include:

[0112] S401: The client 300 presents a configuration interface, which is used to prompt the user to configure the recording task, detection algorithm, recording requirements and presentation format.

[0113] Among them, the configuration interface presented by the client 300 can be provided by the acquisition module 201. The configuration interface can present multiple candidate item identifiers for recording tasks, detection algorithms, recording requirements and expression forms, so that the user can complete the corresponding configuration by selecting the multiple candidate item identifiers.

[0114] S402: Based on the user's configuration operation, the client 300 sends the configuration information to the acquisition module 201, and the configuration information includes the identifier of the hand motion acquisition task, the identifier of the detection algorithm for detecting hand motions, the identifier of the recording requirements, and the identifier of the expression form. The client 300 also provides the identifier of the detection algorithm to the detection module 202, and provides the identifier of the recording requirements and the identifier of the expression form to the output module 203.

[0115] S403: The acquisition module 201 instructs the client 300 to provide a video stream of the hand movements of the object to be captured according to the identifier of the hand movement capture task in the received configuration information.

[0116] S404: The client 300 instructs the collection device 100 to provide a recorded video of the hand movements of the collected object.

[0117] S405: The acquisition device 100 records the hand movements of the object being acquired to obtain a corresponding video stream, and provides the recorded video stream to the client 300.

[0118] During the video recording process, the client 300 may present corresponding prompt information to the subject, so that the subject makes corresponding hand movements according to the prompt information. Alternatively, the subject may make corresponding hand movements under the guidance of the user.

[0119] S406 : the client 300 forwards the video stream to the acquisition module 201 , and the acquisition module 201 provides the video stream to the detection module 202 .

[0120] S407: The detection module 202 accesses the detection algorithm configured by the user and the detection configuration corresponding to the detection algorithm from the database according to the identification of the detection algorithm.

[0121] S408: The detection module 202 detects the hand movements in the video stream using the detection algorithm and the detection configuration, obtains the detection result, and provides the detection result to the output module 203.

[0122] Exemplarily, the detection result includes multiple hand movements in the video stream and the duration of each hand movement.

[0123] S409: The output module 203 determines whether the detection result meets the recording requirement.

[0124] Exemplarily, the recording requirement is specifically used to constrain the duration of each hand movement of the collected object to reach a preset duration. Then, the output module 203 can determine each hand movement made by the collected object according to the detection result, and for each hand movement, the output module 203 further determines whether the duration of the hand movement reaches the preset duration. If yes, it indicates that the hand movement made by the collected object during the video recording meets the requirements; if not, it indicates that the hand movement made by the collected object during the video recording does not meet the requirements. For example, the hand movement of the collected object is a "fist-making" movement, and the recording requirement configured by the user indicates that the duration of the "fist-making" movement is not more than 5 seconds and not less than 2 seconds. Then, when the output module 203 determines that the duration of the "fist-making" movement is less than 2 seconds or greater than 5 seconds, it is determined that the hand movement does not meet the recording requirements, and when the output module 203 determines that the duration of the "fist-making" movement is between 2 seconds and 5 seconds, it is determined that the hand movement meets the recording requirements. For another example, the recording requirement is that before the hand movement of the collected object changes from the "stationary state" to the "manual state", the duration of maintaining the "stationary state" is not less than 2 seconds. After detecting the change in the hand movement of the collected object, the output module 203 can detect whether the duration of the "stationary state" of the collected object before the hand movement changes is less than 2 seconds; if it is less, the output module 203 determines that the duration of the "stationary state" maintained by the hand of the collected object does not meet the recording requirement; if it is not less, the output module 203 determines that the hand movement of the collected object meets the recording requirement.

[0125] S410: When the detection result does not meet the recording requirement, the output module 203 generates a recording suggestion for the first expression form of the captured object.

[0126] For example, when the duration of the "fist clenching" action is less than 2 seconds or greater than 5 seconds, the output module 203 can generate a recording suggestion including a "fist clenching" action identifier and a duration of 2 to 5 seconds, so as to instruct the video of the "fist clenching" action of the captured object to be re-recorded, and the re-recorded "fist clenching" action is an action with a duration within the range of 2 to 5 seconds.

[0127] The output module 203 generates the recording suggestion in the first expression form by default. The first expression form may be, for example, a text-type recording suggestion, such as the recording suggestion specifically including text-type information such as “make a fist” and “2 seconds to 5 seconds”.

[0128] In this embodiment, the output module 203 can output the recording suggestion in the form desired by the user. Specifically, Figure 4 The method shown may also include the following steps.

[0129] S411: The output module 203 transforms the recording suggestion in the first expression form into the recording suggestion in the second expression form according to the expression form configured by the user.

[0130] In this embodiment, the second expression form can be an expression form that can improve the friendliness of human-computer interaction, such as an expression form of pictures, videos, voices, etc. For example, when the recording suggestion of the first expression form is specifically "make a fist" or "2 seconds to 5 seconds", the recording suggestion of the second expression form generated by the output module 203 can be specifically the audio corresponding to "please make a fist again, and the fist action is about 3 seconds", or it can be a picture / video including the fist action and the duration. In this way, the user / collected object can learn about the recording suggestion in a more convenient or more preferred way, thereby improving the interactivity of human-computer interaction.

[0131] S412: The output module 203 outputs the recording suggestion in the second representation form to the client 300.

[0132] S413: The output module 203 outputs the processed video stream to the client 300, where the video stream includes the markers of each hand movement detected by the detection module 202.

[0133] In this way, the captured object can reuse the capture device 100 to perform the recording process for hand movements that do not meet the recording requirements based on the recording suggestions and video streams output by the processing device 200. For the re-recorded video stream, the above-mentioned step S405 and subsequent steps can be re-executed until the hand movements of the captured object in the recorded video stream meet the recording requirements configured by the user. Among them, the re-recorded video stream can be a video stream obtained by re-recording only the hand movements that do not meet the recording requirements, so that the captured object does not need to re-record all the hand movements, thereby improving the overall hand movement recording efficiency.

[0134] It is worth noting that Figure 4 The method embodiment shown is only used as an exemplary description. In other possible embodiments, the output module 203 can also output the recording suggestion to the acquisition device 100, so that the collected object can directly determine the hand movements that do not meet the recording requirements on the collected object 100, so that the collected object can use the acquisition device 100 to perform the process of re-recording the hand movements.

[0135] in addition, Figure 4 For the specific implementation of each step in the method embodiment shown, please refer to the aforementioned Figure 2 The description of the relevant parts in the illustrated embodiment will not be repeated here.

[0136] See also Figure 5, shows a flow chart of another method for processing a video stream. Figure 5 In the method embodiment shown, the recording task may specifically be a task of recording a video of the facial expression of the subject being collected. The facial expression may be, for example, an expression such as "open mouth" or "closed eyes". Figure 5 As shown, the method may specifically include:

[0137] S501: The client 300 presents a configuration interface, which is used to prompt the user to configure the recording task, detection algorithm, recording requirements and presentation form.

[0138] S502: Based on the user's configuration operation, the client 300 sends the configuration information to the acquisition module 201, and the configuration information includes the identification of the facial expression acquisition task, the identification of the detection algorithm for detecting facial expressions, the identification of the recording requirements, and the identification of the expression form. The identification of the detection algorithm is provided to the detection module 202, and the identification of the recording requirements and the identification of the expression form are provided to the output module 203.

[0139] S503: The acquisition module 201 instructs the client 300 to provide a video stream of the facial expression of the object to be collected according to the identification of the facial expression collection task in the received configuration information.

[0140] S504: The client 300 instructs the collection device 100 to provide a recorded video of the facial expression of the collected object.

[0141] S505: The collection device 100 records the facial expression of the collected object to obtain a corresponding video stream, and provides the recorded video stream to the client 300.

[0142] S506 : the client 300 forwards the video stream to the acquisition module 201 , and the acquisition module 201 provides the video stream to the detection module 202 .

[0143] S507: The detection module 202 accesses the detection algorithm configured by the user and the detection configuration corresponding to the detection algorithm from the database according to the identification of the detection algorithm.

[0144] S508 : The detection module 202 detects the facial expression in the video stream using the detection algorithm and the detection configuration, obtains the detection result, and provides the detection result to the output module 203 .

[0145] Exemplarily, the detection result includes multiple facial expressions in the video stream and the movement amplitude of each facial expression.

[0146] S509: The output module 203 determines whether the detection result meets the recording requirement.

[0147] Exemplarily, the recording requirement is specifically used to constrain the movement amplitude of each facial expression of the collected object to be greater than a preset amplitude. Then, the output module 203 can determine each facial expression made by the collected object according to the detection result, and for each facial expression, the output module 203 further determines whether the facial expression is greater than the preset amplitude. If yes, it indicates that the facial expression made by the collected object during the video recording meets the requirement; if not, it indicates that the facial expression made by the collected object during the video recording does not meet the requirement. For example, the facial expression of the collected object is an "open mouth" expression, and the recording requirement configured by the user indicates that the amplitude of the mouth opening in the "open mouth" expression is greater than a threshold value 1. Then, when the output module 203 determines that the amplitude of the mouth opening in the "open mouth" expression is less than or equal to the threshold value 1, it is determined that the facial expression does not meet the recording requirement, and when the output module 203 determines that the amplitude of the mouth opening in the "open mouth" expression is greater than the threshold value 1, it is determined that the facial expression meets the recording requirement.

[0148] S510: When the detection result does not meet the recording requirement, the output module 203 generates a recording suggestion for the first expression form of the captured object.

[0149] For example, when the degree of mouth opening in the "open mouth" expression is less than a threshold value 1, the output module 203 can generate a recording suggestion including an "open mouth" expression logo and increasing the degree of mouth opening, so as to instruct a video to re-record the "open mouth" expression of the captured object, and the re-recorded "open mouth" expression is specifically an expression with the mouth opened with a larger degree.

[0150] The output module 203 generates the recording suggestion in the first expression form by default. The first expression form may be, for example, a text-type recording suggestion, such as the recording suggestion specifically including text-type information such as "open your mouth" or "please open your mouth wider".

[0151] S511: The output module 203 transforms the recording suggestion in the first expression form into the recording suggestion in the second expression form according to the expression form configured by the user.

[0152] In this embodiment, the second expression form can be an expression form that can improve the friendliness of human-computer interaction, such as an expression form of pictures, videos, voices, etc. For example, when the recording suggestion of the first expression form is specifically "open your mouth" or "please open your mouth wider", the recording suggestion of the second expression form generated by the output module 203 can be specifically the audio corresponding to "please make the mouth-opening expression again, open your mouth wider", or it can be a picture / video including the mouth-opening expression and the mouth-opening amplitude. In this way, the user / collected object can learn about the recording suggestion in a more convenient or more preferred way, thereby improving the interactivity of human-computer interaction.

[0153] S512: The output module 203 outputs the recording suggestion in the second representation form to the client 300.

[0154] S513 : the output module 203 outputs the processed video stream to the client 300 , where the video stream includes the markers of each facial expression detected by the detection module 202 .

[0155] Figure 5 For the specific implementation of each step in the method embodiment shown, please refer to the aforementioned Figure 2 The description of the relevant parts in the illustrated embodiment will not be repeated here.

[0156] See also Figure 6 , shows a flow chart of another method for processing a video stream. Figure 6 In the method embodiment shown, the recording task may specifically be a task of redirecting the body movements of the captured object to a virtual character. Figure 6 As shown, the method may specifically include:

[0157] S601: The client 300 presents a configuration interface, which is used to prompt the user to configure the recording task, detection algorithm, recording requirements and presentation format.

[0158] S602: Based on the user's configuration operation, the client 300 sends the configuration information to the acquisition module 201. The configuration information includes an identifier of the action redirection task, an identifier of a detection algorithm for detecting action problems during the redirection process, an identifier of the recording requirement, and an identifier of the expression form. The client 300 provides the identifier of the detection algorithm to the detection module 202, and provides the identifier of the recording requirement and the identifier of the expression form to the output module 203.

[0159] S603: The acquisition module 201 instructs the client 300 to provide a video stream of the body movements of the captured object according to the identifier of the movement redirection task in the received configuration information.

[0160] S604: The client 300 instructs the collection device 100 to provide a recorded video of the body movements of the collected object.

[0161] S605: The collection device 100 records the body movements of the collected object to obtain a corresponding video stream, and provides the recorded video stream to the client 300.

[0162] S606 : the client 300 forwards the video stream to the acquisition module 201 , and the acquisition module 201 provides the video stream to the detection module 202 .

[0163] S607: The detection module 202 accesses the detection algorithm configured by the user and the detection configuration corresponding to the detection algorithm from the database according to the identification of the detection algorithm.

[0164] S608: The detection module 202 redirects the body movements of the captured object in the video stream into the body movements of the virtual character, thereby generating a video stream of the virtual character.

[0165] The body movements of the virtual character in the video stream of the virtual character are consistent with the body movements of the captured object, thereby redirecting the body movements of the captured object to the virtual character.

[0166] S609: The detection module 202 detects the action problem in the video stream of the virtual character using the detection algorithm and the detection configuration, obtains the detection result, and provides the detection result to the output module 203.

[0167] Exemplarily, the detection result includes detailed information such as the motion problems existing in the video stream of the virtual character (such as limb penetration, etc.) and the location of the motion problems in the video stream.

[0168] S610: The output module 203 determines whether the detection result meets the recording requirement.

[0169] For example, the recording requirement is specifically used to constrain the video stream of the virtual character to not have a model penetration problem. Then, the output module 203 can determine whether there is a model penetration in the video stream of the virtual character according to the detection result. If yes, the video stream representing the virtual character does not meet the requirement; if not, the video stream representing the virtual character meets the requirement.

[0170] S611: When the detection result does not meet the recording requirement, the output module 203 generates a recording suggestion for the first representation form of the captured object.

[0171] For example, when there is a problem of penetration between the arm and torso in the video stream of the virtual character, the output module 203 can generate a recording suggestion including an "arm" mark and raising the arm, so as to instruct the re-recording of the video of the captured object's limb movement of the "arm", and the re-recorded limb movement is specifically the action of "raising the arm".

[0172] Among them, the output module 203 generates recording suggestions in the first form by default. The first form can be, for example, a text-type recording suggestion, such as the recording suggestion specifically including text-type information such as "There is penetration between the arm and the torso" and "Please raise your arms".

[0173] S612: The output module 203 transforms the recording suggestion in the first expression form into the recording suggestion in the second expression form according to the expression form configured by the user.

[0174] In this embodiment, the second expression form can be an expression form that can improve the friendliness of human-computer interaction, such as an expression form of pictures, videos, voices, etc. For example, when the recording suggestion of the first expression form is specifically "there is a model penetration between the arm and the torso" or "please raise your arm", the recording suggestion of the second expression form generated by the output module 203 can specifically be the audio corresponding to "please raise your arm". In this way, the user / collected object can know the recording suggestion in a more convenient or more preferred way, thereby improving the interactivity of human-computer interaction.

[0175] S613: The output module 203 outputs the recording suggestion in the second representation form to the client 300.

[0176] S614: The output module 203 outputs the processed video stream of the virtual character to the client 300, where the video stream includes the marks of each body movement detected by the detection module 202 and the movement problems existing in some body movements.

[0177] Figure 6 For the specific implementation of each step in the method embodiment shown, please refer to the aforementioned Figure 2 The description of the relevant parts in the illustrated embodiment will not be repeated here.

[0178] See also Figure 7 , shows a flowchart of a method for processing an audio stream. Figure 7 In the method embodiment shown, the recording task may specifically be a task of recording the audio of the collected object's voice. Figure 7 As shown, the method may specifically include:

[0179] S701: The client 300 presents a configuration interface, which is used to prompt the user to configure the recording task, detection algorithm, recording requirements and presentation format.

[0180] S702: Based on the user's configuration operation, the client 300 sends the configuration information to the acquisition module 201, and the configuration information includes the identifier of the voice collection task, the identifier of the detection algorithm for detecting action problems in the redirection process, the identifier of the recording requirement, and the identifier of the expression form. The client 300 provides the identifier of the detection algorithm to the detection module 202, and provides the identifier of the recording requirement and the identifier of the expression form to the output module 203.

[0181] In actual application, the user's configuration of recording requirements can also be expressed as a configuration of application scenarios for voice collection. At this time, according to different application scenarios configured by the user, the collection device 100 can use different recording texts to perform audio recording of the voice of the collected object.

[0182] S703: The acquisition module 201 instructs the client 300 to provide an audio stream of the voice of the collected object according to the identifier of the voice collection task in the received configuration information.

[0183] S704: The client 300 instructs the collection device 100 to provide recorded audio of the voice of the collected object.

[0184] S705: The collection device 100 records the voice of the collected object to obtain a corresponding audio stream, and provides the recorded audio stream to the client 300.

[0185] Among them, the collected device can output voice based on a specific recording text, and the recording text can be determined according to the recording requirements (ie, application scenarios) configured by the user.

[0186] S706 : the client 300 forwards the audio stream to the acquisition module 201 , and the acquisition module 201 provides the audio stream to the detection module 202 .

[0187] S707: The detection module 202 accesses the detection algorithm configured by the user and the detection configuration corresponding to the detection algorithm from the database according to the identification of the detection algorithm.

[0188] S708: The detection module 202 detects the voice of the collected object in the audio stream, obtains a detection result, and provides the detection result to the output module 203.

[0189] Exemplarily, the detection result includes the speech meaning included in the audio stream, and the speech meaning may be, for example, the ambient noise, echo, pronunciation of the collected object, speech emotion, and speech style in the audio stream. In addition, the detection result may also include attribute information corresponding to the speech meaning, such as the decibel of the ambient noise.

[0190] S709: The output module 203 determines whether the detection result meets the recording requirement.

[0191] Illustratively, the recording requirements are specifically used to constrain the meaning of speech in the audio stream.

[0192] When the speech meaning is specifically environmental noise, the output module 203 may specifically determine whether the decibel of the environmental noise in the audio stream is greater than a threshold value 1, that is, determine whether the recording environment of the collected object is quiet.

[0193] When the speech meaning is specifically an echo, the output module 203 may specifically determine whether the decibel of the echo in the audio stream is greater than a threshold 2, that is, determine whether the collected object will generate an echo that affects the recording quality during recording.

[0194] When the speech meaning is pronunciation, the output module 203 may specifically determine whether the pronunciation of the collected object conforms to the standard pronunciation corresponding to the recording text, that is, determine whether the pronunciation of the collected object has problems such as inaccurate pronunciation or unclear pronunciation.

[0195] When the speech meaning is specifically speech emotion, the output module 203 may specifically determine whether the speech emotion corresponding to the pronunciation of the collected object in the audio stream is a preset emotion, such as determining whether the pronunciation of the collected object includes the tone of the preset emotion.

[0196] When the speech meaning is specifically a speech style, the output module 203 may specifically determine whether the speech style corresponding to the pronunciation of the collected object in the audio stream is a preset style, such as determining whether the pronunciation of the collected object includes the tone of the preset style.

[0197] S710: When the detection result does not meet the recording requirement, the output module 203 generates a recording suggestion for the first expression form of the captured object.

[0198] For example, when the pronunciation of the collected object in the audio stream lacks obvious expression of emotion / style, the output module 203 can generate a recording suggestion including an "emotion" identifier / "style" identifier to instruct the re-recording of the audio of the collected object for a specific emotion / specific style, and the emotion / style of the re-recorded audio pronunciation is the preset emotion / style.

[0199] The output module 203 generates the recording suggestion in a first expression form by default. The first expression form may be, for example, a text-type recording suggestion, such as a text-type information such as “Please record in XX style”.

[0200] S711: The output module 203 transforms the recording suggestion in the first expression form into the recording suggestion in the second expression form according to the expression form configured by the user.

[0201] In this embodiment, the second expression form can be an expression form that can improve the friendliness of human-computer interaction, such as an expression form of pictures, videos, voices, etc. For example, when the recording suggestion of the first expression form is specifically a text-type information "Please record in XX style", the recording suggestion of the second expression form generated by the output module 203 can specifically be the audio corresponding to "Please record in XX style". In this way, the user / collected object can know the recording suggestion in a more convenient or more preferred way, thereby improving the interactivity of human-computer interaction.

[0202] S712: The output module 203 outputs the recording suggestion in the second representation form to the client 300.

[0203] S713: The output module 203 outputs the processed audio stream to the client 300, where the audio stream includes the detection result detected by the detection module 202.

[0204] Figure 7 For the specific implementation of each step in the method embodiment shown, please refer to the aforementioned Figure 2 The description of the relevant parts in the illustrated embodiment will not be repeated here.

[0205] Above Figures 1 to 7 In the illustrated embodiment, the processing device 200 may be software configured on a computing device or a computing device cluster, and by running the software on the computing device or the computing device cluster, the computing device or the computing device cluster may implement the functions of the processing device 200. The processing device 200 is described in detail below from the perspective of hardware device implementation.

[0206] Figure 8 The schematic diagram of the structure of a computing device is shown. The processing device 200 can be deployed on the computing device. The computing device can be a computing device in a cloud environment (such as a server) or a computing device in an edge environment. Specifically, it can be used to implement the above Figure 2 , Figures 4 to 7 The functionality of the processing device 200 in the illustrated embodiment.

[0207] like Figure 8 As shown, computing device 800 includes processor 810, memory 820, communication interface 830 and bus 840. Processor 810, memory 820 and communication interface 830 communicate through bus 840. Bus 840 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The communication interface 830 is used to communicate with the outside, for example, to obtain a first video stream, send a second video stream, etc.

[0208] Among them, the processor 810 can be a central processing unit (CPU), an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits. The processor 810 can also be an integrated circuit chip with signal processing capabilities. In the implementation process, the functions of each device in the processing device 200 can be completed by the hardware integrated logic circuit or software instructions in the processor 810. The processor 810 can also be a general processor, a digital signal processor (DSP), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. Among them, the general processor can be a microprocessor or the processor can also be any conventional processor, etc., and the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 820 , and the processor 810 reads the information in the memory 820 and completes part or all of the functions in the processing device 200 in combination with its hardware.

[0209] The memory 820 may include a volatile memory, such as a random access memory (RAM), or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a HDD, or a SSD.

[0210] The memory 820 stores executable codes, and the processor 810 executes the executable codes to execute the method executed by the aforementioned processing device 200 .

[0211] Specifically, in implementing Figure 2 , Figures 4 to 7 In the case of the embodiment shown, and Figure 2 , Figures 4 to 7 In the case where the processing device 200 described in the illustrated embodiment is implemented by software, the execution Figure 2 , Figures 4 to 7 The software or program code required for the functions of the processing device 200 is stored in the memory 820. The interaction between the processing device 200 and the acquisition device 100 is realized through the communication interface 830. The processor is used to execute the instructions in the memory 820 to implement the method executed by the processing device 200.

[0212] Fig. 9 A schematic diagram of the structure of a computing device cluster is shown. Fig. 9 The computing device cluster 90 shown includes multiple computing devices, and the processing device 200 can be deployed on multiple computing devices in the computing device cluster 90 in a distributed manner. Fig. 9 As shown, the computing device cluster 90 includes multiple computing devices 900 , each computing device 900 includes a memory 920 , a processor 910 , a communication interface 930 and a bus 940 , wherein the memory 920 , the processor 910 , and the communication interface 930 are communicatively connected to each other via the bus 940 .

[0213] The processor 910 may be a CPU, a GPU, an ASIC, or one or more integrated circuits. The processor 910 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, some functions of the processing device 200 may be completed by the hardware integrated logic circuit or software instructions in the processor 910. The processor 910 may also be a DSP, an FPGA, a general processor, other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and may implement or execute some methods, steps, and logic block diagrams disclosed in the embodiments of the present application. Among them, the general processor may be a microprocessor or the processor may also be any conventional processor, etc., and the steps of the method disclosed in the embodiments of the present application may be directly embodied as a hardware decoding processor for execution, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 920. In each computing device 900, the processor 910 reads the information in the memory 920, and in combination with its hardware, some functions of the processing device 200 may be completed.

[0214] The memory 920 may include ROM, RAM, static storage device, dynamic storage device, hard disk (e.g., SSD, HDD), etc. The memory 920 may store program codes, for example, part or all of the program codes for implementing the acquisition module 201, part or all of the program codes for implementing the detection module 202, part or all of the program codes for implementing the output module 203, etc. For each computing device 900, when the program code stored in the memory 920 is executed by the processor 910, the processor 910 executes part of the method executed by the processing device 200 based on the communication interface 930. The memory 920 may also store data, for example: intermediate data or result data generated by the processor 910 during the execution process, for example, the above-mentioned detection results, recording suggestions, etc.

[0215] The communication interface 903 in each computing device 900 is used for external communication, such as interacting with other computing devices 900 .

[0216] The bus 940 may be a peripheral component interconnect standard bus or an extended industry standard architecture bus. For ease of representation, Fig. 9 The bus 940 in each computing device 900 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0217] The plurality of computing devices 900 establish communication paths through a communication network to implement the functions of the processing apparatus 200. Any computing device may be a computing device (eg, a server) in a cloud environment, or a computing device in an edge environment.

[0218] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on one or more computing devices, the one or more computing devices execute the method executed by the processing device 200 of the above embodiment.

[0219] In addition, the embodiment of the present application further provides a computer program product, and when the computer program product is executed by one or more computing devices, the one or more computing devices execute any of the aforementioned methods for processing multimedia streams. The computer program product may be a software installation package, and when any of the aforementioned methods for processing multimedia streams is required, the computer program product may be downloaded and executed on a computer.

[0220] It should also be noted that the system embodiments described above are merely schematic, wherein the devices described as separate components may or may not be physically separated, and the components displayed as devices may or may not be physical devices, that is, they may be located in one place, or they may be distributed on multiple network devices. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the system embodiments provided by the present application, the connection relationship between the devices indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0221] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0222] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0223] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for processing multimedia streams, characterized in that: The method comprises: Output a configuration interface, wherein the configuration interface is used to prompt configuration of recording tasks, detection algorithms, and recording requirements; In response to an operation of a user on the configuration interface, obtaining first configuration information provided by the user, where the first configuration information is used to configure a recording task, a detection algorithm, and a recording requirement; Acquire a multimedia stream under the recording task, wherein the multimedia stream includes a video stream or an audio stream, wherein the video stream is obtained by recording a video of the object to be captured, and the audio stream is obtained by recording a voice of the object to be captured; According to the detection algorithm, the multimedia stream is detected to obtain a detection result; When the detection result does not meet the recording requirement, a recording suggestion for the captured object is output.

2. The method according to claim 1, characterized in that: The recording requirement is used to constrain requirements to be met by video content in the video stream obtained by recording, or is used to constrain requirements to be met by audio content in the audio stream obtained by recording.

3. The method according to claim 1 or 2, characterized in that: The recording task includes a limb motion collection task, the multimedia stream is a video stream, the detection algorithm is used to detect limb motions in the video stream, and the recording requirements are used to constrain requirements satisfied by the limb motions in the video stream.

4. The method according to claim 1 or 2, characterized in that: The recording task includes an expression collection task, the multimedia stream is a video stream, the detection algorithm is used to detect facial expressions in the video stream, and the recording requirements are used to constrain requirements satisfied by facial expressions in the video stream.

5. The method according to claim 1 or 2, characterized in that: The recording task includes an action redirection task, the multimedia stream is a video stream, the detection algorithm is used to detect action problems existing in redirecting body movements in the video stream to a virtual character, and the recording requirements are used to constrain the requirements met by the body movements in the video stream redirected to the virtual character.

6. The method according to claim 1 or 2, characterized in that: The recording task includes a voice collection task, the multimedia stream is an audio stream, the detection algorithm is used to detect the voice meaning in the audio stream, and the recording requirement is used to constrain the requirements met by the voice meaning in the audio stream.

7. The method according to any one of claims 1 to 6, characterized in that: The configuration interface is also used to prompt the configuration of the presentation form of the recording suggestion, and the method further includes: In response to an operation of the user on the configuration interface, obtaining second configuration information of a presentation form of the recording suggestion provided by the user; When the detection result does not meet the recording requirement, outputting a recording suggestion for the captured object includes: When the detection result does not meet the recording requirement, generating a recording suggestion in a first expression form; According to the second configuration information, the recording suggestion in the first expression form is transformed to obtain the recording suggestion in the second expression form; Output a recording suggestion in the second expression form.

8. The method according to claim 7, characterized in that The acquiring, in response to the user's operation on the configuration interface, second configuration information for the presentation form of the recording suggestion provided by the user includes: Acquire a second configuration file input by the user on the configuration interface, where the second configuration file includes the second configuration information; Alternatively, in response to the user's selection operation on identification of multiple candidate representation forms, the second configuration information configured by the user is obtained.

9. The method according to any one of claims 1 to 8, characterized in that: The step of obtaining, in response to an operation of the user on the configuration interface, first configuration information provided by the user includes: In response to a selection operation by the user on the configuration interface for identifications of a plurality of candidate recording tasks, determining a recording task configured by the user; In response to the user's selection operation on the configuration interface for identifications of multiple candidate detection algorithms, determining the detection algorithm configured by the user; In response to the user's selection operation on the configuration interface for identification of multiple candidate recording requirements, the recording requirement configured by the user is determined.

10. The method according to claim 9, characterized in that The multiple candidate detection algorithms and the multiple candidate recording requirements are stored in a database, and the method further includes: Outputting an update interface, wherein the update interface is used to prompt to update the database; In response to a user's update operation on the database on the update interface, the candidate detection algorithm or the candidate recording requirement in the database is updated.

11. The method according to any one of claims 1 to 8, characterized in that: The step of obtaining, in response to an operation of the user on the configuration interface, first configuration information provided by the user includes: A first configuration file input by a user on the configuration interface is obtained, where the first configuration file includes the recording task, the detection algorithm, or the recording requirement.

12. The method according to any one of claims 1 to 11, characterized in that: The multimedia stream includes a video stream, and the method further includes: Merging the detection result or the recording suggestion with the video stream to obtain a fused video stream; The fused video stream is output.

13. A multimedia stream processing device, characterized in that: The device comprises: an acquisition module, configured to output a configuration interface, wherein the configuration interface is used to prompt the user to configure a recording task, a detection algorithm, and a recording requirement; in response to an operation of a user on the configuration interface, acquire first configuration information provided by the user, wherein the first configuration information is used to configure the recording task, the detection algorithm, and the recording requirement; acquire a multimedia stream under the recording task, wherein the multimedia stream includes a video stream or an audio stream, wherein the video stream is obtained by recording a video of a captured object, and the audio stream is obtained by recording a voice of a captured object; A detection module, used to detect the multimedia stream according to the detection algorithm to obtain a detection result; The output module is used to output a recording suggestion for the captured object when the detection result does not meet the recording requirement.

14. The device according to claim 13, characterized in that The recording requirement is used to constrain requirements to be met by video content in the video stream obtained by recording, or is used to constrain requirements to be met by audio content in the audio stream obtained by recording.

15. The device according to claim 13 or 14, characterized in that The recording task includes a limb motion collection task, the multimedia stream is a video stream, the detection algorithm is used to detect limb motions in the video stream, and the recording requirements are used to constrain requirements satisfied by the limb motions in the video stream.

16. The device according to claim 13 or 14, characterized in that The recording task includes an expression collection task, the multimedia stream is a video stream, the detection algorithm is used to detect facial expressions in the video stream, and the recording requirements are used to constrain requirements satisfied by facial expressions in the video stream.

17. The device according to claim 13 or 14, characterized in that The recording task includes an action redirection task, the multimedia stream is a video stream, the detection algorithm is used to detect action problems existing in redirecting body movements in the video stream to a virtual character, and the recording requirements are used to constrain the requirements met by the body movements in the video stream redirected to the virtual character.

18. The device according to claim 13 or 14, characterized in that The recording task includes a voice collection task, the multimedia stream is an audio stream, the detection algorithm is used to detect the voice meaning in the audio stream, and the recording requirement is used to constrain the requirements met by the voice meaning in the audio stream.

19. The device according to any one of claims 13 to 18, characterized in that The configuration interface is also used to prompt configuration of the presentation form of the recording suggestion; The acquisition module is further configured to acquire, in response to an operation of the user on the configuration interface, second configuration information for the expression form of the recording suggestion provided by the user; The output module is used for: When the detection result does not meet the recording requirement, generating a recording suggestion in a first expression form; According to the second configuration information, the recording suggestion in the first expression form is transformed to obtain the recording suggestion in the second expression form; Output a recording suggestion in the second expression form.

20. The device according to claim 19, characterized in that The acquisition module is used to: Acquire a second configuration file input by the user on the configuration interface, where the second configuration file includes the second configuration information; Alternatively, in response to the user's selection operation on identification of multiple candidate representation forms, the second configuration information configured by the user is obtained.

21. The device according to any one of claims 13 to 20, characterized in that The acquisition module is used to: In response to a selection operation by the user on the configuration interface for identifications of a plurality of candidate recording tasks, determining a recording task configured by the user; In response to the user's selection operation on the configuration interface for identifications of multiple candidate detection algorithms, determining the detection algorithm configured by the user; In response to the user's selection operation on the configuration interface for identification of multiple candidate recording requirements, the recording requirement configured by the user is determined.

22. The device according to claim 21, characterized in that The multiple candidate detection algorithms and the multiple candidate recording requirements are stored in a database, and the acquisition module is further used to: Outputting an update interface, wherein the update interface is used to prompt to update the database; In response to a user's update operation on the database on the update interface, the candidate detection algorithm or the candidate recording requirement in the database is updated.

23. The device according to any one of claims 13 to 20, characterized in that The acquisition module is used to acquire a first configuration file input by a user on the configuration interface, where the first configuration file includes the recording task, the detection algorithm, or the recording requirement.

24. The device according to any one of claims 13 to 23, characterized in that The multimedia stream includes a video stream, and the output module is further used for: Merging the detection result or the recording suggestion with the video stream to obtain a fused video stream; The fused video stream is output.

25. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the computing device cluster executes the method according to any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on at least one computing device, enable the at least one computing device to perform the method according to any one of claims 1 to 12.

Images