Multimedia stream processing method and apparatus, and related device

Abstract

Provided in the present application is a multimedia stream processing method. The method comprises: outputting a configuration interface, and in response to a user's operation on the configuration interface, acquiring first configuration information, wherein the first configuration information is used for configuring a recording task, a test algorithm and recording requirements; acquiring a multimedia stream under the recording task, wherein the multimedia stream is a video stream that is obtained by means of performing video recording on a captured object, or an audio stream that is obtained by means of performing voice recording thereon; and on the basis of the test algorithm configured by the user, testing the multimedia stream, so as to obtain a test result, and when the test result does not meet the recording requirements configured by the user, outputting a recording suggestion for the captured object. In this way, by means of feeding back a suggestion, the efficiency of generating a recorded video / audio, which meets the expectation of a user, can be effectively improved, and the differentiated requirements of different users for different recording tasks, different test algorithms and different recording requirements can be met. In addition, further provided in the present application are a corresponding processing apparatus and a related device.

Description

Multimedia stream processing method, device and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 11, 2023, with application number 202311319379.8 and application name “Method, device and related equipment for processing multimedia streams”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 27, 2023, with application number 202311412575.X and application name “Method, device and related equipment for processing multimedia streams”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of multimedia technology, and in particular to a method, apparatus and related equipment for processing multimedia streams. Background Art

[0003] Currently, video and audio recording are widely used in numerous business scenarios. For example, digital humans are digitized human figures created using digital technology that closely resemble human forms. The process of generating digital humans typically involves recording real human expressions and body movements to create a digital human capable of expressing emotions and actions. Alternatively, real human speech can be recorded to create a digital human capable of producing human speech.

[0004] Currently, after completing a video recording, users typically re-watch the video. If the video has flaws, such as inadequate body movements, the user will re-record the video. This process of repeated recording and viewing eventually results in a video that meets the user's expectations. Similarly, users can repeatedly record and listen to audio to achieve the desired audio. However, this method of generating videos or audio that meet user expectations is inefficient and labor-intensive.

[0005] Summary of the Invention

[0006] In view of this, embodiments of the present application provide a method for processing multimedia streams to improve the accuracy of determining the root cause of anomalies for an object. The present application also provides a corresponding data processing apparatus, a computing device cluster, a computer-readable storage medium, and a computer program product.

[0007] In a first aspect, an embodiment of the present application provides a method for processing a multimedia stream. The method can be executed by a corresponding processing device. Specifically, the processing device outputs a configuration interface, such as a configuration interface that can be output to a client provided to the outside. The configuration interface is used to prompt the configuration of a recording task, a detection algorithm, and recording requirements. The recording task can be, for example, a video recording task for changes in body movements, a video recording task for changes in facial expressions, a video recording task for action redirection, an audio recording task for the voice of a person or animal, etc. The detection algorithm can be, for example, an algorithm for detecting body movements expressed in multiple frames of a video stream; and the recording requirement can be, for example, a requirement that a hand movement remain in a "still" state for a duration of not less than 2 seconds. Then, the processing device, in response to a user's operation on the configuration interface, obtains first configuration information provided by the user. The first configuration information is used to configure the recording task, the detection algorithm, and the recording requirements. Next, the processing device obtains the multimedia stream under the recording task configured by the user. The multimedia stream can be a video stream or an audio stream. The video stream is obtained by video recording the captured object, and the audio stream is obtained by recording the captured object's voice. Finally, the processing device detects the multimedia stream using the user-configured detection algorithm, obtains a detection result, and determines whether the detection result meets the user-configured recording requirements. If the detection result does not meet the user-configured recording requirements, the processing device outputs a recording suggestion for the subject, such as suggesting that the subject re-record part of the content.

[0008] Since the processing device can automatically process the recorded video stream / audio stream and output recording suggestions according to the recording requirements, the user does not need to repeatedly watch the video or listen to the audio to determine the problems in the video / audio, but can directly re-record the video or audio based on the output recording suggestions, thereby effectively improving the efficiency of generating recorded video / audio that meets the user's expectations. In actual application, during the process of recording video or audio, recording suggestions for the current recording stage can be output in real time according to the recording requirements. This allows the user to re-record only the current recording stage without having to perform the re-recording process of all recording stages, which can further improve the efficiency of generating recorded video / audio. Moreover, the processing device can support the user to independently configure the recording tasks, detection algorithms and recording requirements. This can meet the differentiated needs of different users for different recording tasks, different detection algorithms and different recording requirements, thereby improving the flexibility of users in generating recorded video / audio and enriching the application scenarios applicable to audio and video recording.

[0009] In one possible implementation, the user-configured recording requirements are used to constrain the requirements that the video content in the recorded video stream must meet, or to constrain the requirements that the audio content in the recorded audio stream must meet. This allows users to customize their video or audio recording requirements to meet their needs in different business scenarios (i.e., for the same type of recording task, users may have different requirements for the recorded video or audio in different scenarios).

[0010] In one possible implementation, the recording task configured by the user includes a limb motion collection task. In this case, the multimedia stream is a video stream, which is obtained by recording the limb motions (such as hand motions) of the collected object. The detection algorithm configured by the user is used to detect the limb motions in the video stream, and the recording requirements configured by the user are used to constrain the requirements met by the limb motions in the video stream.

[0011] In one possible implementation, the recording task configured by the user includes an expression capture task, the multimedia stream is a video stream, and the video stream is obtained by recording the facial expressions of the captured object. The detection algorithm configured by the user is used to detect facial expressions in the video stream, and the recording requirements configured by the user are used to constrain the requirements that must be met by the facial expressions in the video stream.

[0012] In one possible implementation, the recording task configured by the user includes a motion redirection task, the multimedia stream is a video stream, and the video stream is obtained by recording the limb movements of the captured object. The detection algorithm configured by the user is used to detect motion problems existing in redirecting the limb movements in the video stream to the virtual character, and the recording requirements configured by the user are used to constrain the requirements met by the limb movements redirected to the virtual character in the video stream.

[0013] In one possible implementation, the recording task configured by the user includes a voice collection task, the multimedia stream is an audio stream, and the audio stream is obtained by recording the pronunciation of the collected object. The detection algorithm configured by the user is used to detect the voice meaning (such as pronunciation style, emotion, etc.) in the audio stream, and the recording requirements configured by the user are used to constrain the requirements met by the voice meaning in the audio stream.

[0014] In one possible embodiment, the configuration interface output by the processing device is also used to prompt the user to configure the form of expression of the recording suggestion. Then, the client provided by the processing device to the outside can respond to the user's operation on the configuration interface and obtain the second configuration information for the form of expression of the recording suggestion provided by the user. In this way, when the processing device outputs the recording suggestion for the collected object, it can be that when the detection result does not meet the recording requirements, it will first generate a recording suggestion of the first form of expression. Then, the processing device transforms the recording suggestion of the first form of expression according to the second configuration information configured by the user, obtains the recording suggestion of the second form of expression, and outputs the recording suggestion of the second form of expression. In this way, not only can the diversity of the recording suggestions output by the processing device be enriched, but also the user's selectivity of the form of expression of the recording suggestion can be increased, thereby further improving the user experience.

[0015] In one possible implementation, when the processing device obtains the second configuration information for the presentation format of the recording suggestion provided by the user, it may specifically obtain a second configuration file entered by the user on a configuration interface, where the second configuration file includes the second configuration information. Alternatively, a client provided by the processing device may obtain the user-configured second configuration information in response to a user's selection operation for identifying multiple candidate presentation formats. In this way, the user can configure the presentation format of the recording suggestion, increasing the user's choice of presentation formats for the recording suggestion, thereby further improving the user experience.

[0016] In one possible implementation, when the processing device obtains the first configuration information provided by the user, it may specifically determine the user-configured recording task in response to the user's selection operation on the configuration interface for multiple candidate recording task identifiers; determine the user-configured detection algorithm in response to the user's selection operation on the configuration interface for multiple candidate detection algorithm identifiers; and determine the user-configured recording requirement in response to the user's selection operation on the configuration interface for multiple candidate recording requirement identifiers. In this way, the user can flexibly select the recording task, detection algorithm, and recording requirement on the configuration interface, thereby improving the flexibility of information configuration and enhancing the user experience.

[0017] In one possible implementation, multiple candidate detection algorithms and multiple candidate recording requirements for user selection are stored in a database. The processing device can then display an update interface through the client. This update interface prompts the user to update the database. In response to the user's update operation on the update interface, the processing device can update the candidate detection algorithms or candidate recording requirements in the database. This dynamic data update allows the candidate recording requirements and candidate detection algorithms in the database to adapt to changes in the user's needs for detection algorithms and recording requirements, further improving the user experience.

[0018] In one possible implementation, when the processing device obtains the first configuration information provided by the user, it may specifically obtain a first configuration file entered by the user on a configuration interface. The first configuration file includes a recording task, a detection algorithm, or recording requirements. In this way, the user can configure the recording task, detection algorithm, or recording requirements by importing the configuration file, thereby enabling user customization of the configuration, improving user configuration flexibility and user experience.

[0019] In one possible implementation, if the multimedia stream acquired by the processing device includes a video stream, the processing device may further fuse the detection results or recording suggestions with the video stream to obtain a fused video stream and output the fused video stream. This allows the user to view the processing device's detection of the content in the video stream, or to identify portions of the video stream that need to be re-recorded, thereby subsequently guiding the subject to re-record that portion of the video, thereby improving the user experience.

[0020] In a second aspect, the present application provides a multimedia stream processing device, which includes: an acquisition module for outputting a configuration interface, the configuration interface is used to prompt the configuration of a recording task, a detection algorithm, and recording requirements, and in response to the user's operation on the configuration interface, obtains first configuration information provided by the user, the first configuration information is used to configure the recording task, the detection algorithm, and the recording requirements, and obtains the multimedia stream under the recording task, the multimedia stream includes a video stream or an audio stream, the video stream is obtained by recording the video of the collected object, and the audio stream is obtained by recording the voice of the collected object; a detection module is used to detect the multimedia stream according to the detection algorithm to obtain a detection result; an output module is used to output a recording suggestion for the collected object when the detection result does not meet the recording requirements.

[0021] In a possible implementation, the recording requirement is used to constrain requirements satisfied by video content in a recorded video stream, or is used to constrain requirements satisfied by audio content in a recorded audio stream.

[0022] In a possible implementation, the recording task includes a limb motion collection task, the multimedia stream is a video stream, the detection algorithm is used to detect limb motions in the video stream, and the recording requirements are used to constrain the requirements met by the limb motions in the video stream.

[0023] In a possible implementation, the recording task includes an expression collection task, the multimedia stream is a video stream, the detection algorithm is used to detect facial expressions in the video stream, and the recording requirements are used to constrain requirements met by facial expressions in the video stream.

[0024] In one possible implementation, the recording task includes an action redirection task, the multimedia stream is a video stream, the detection algorithm is used to detect action problems existing when redirecting limb movements in the video stream to a virtual character, and the recording requirements are used to constrain the requirements met by limb movements redirected to the virtual character in the video stream.

[0025] In a possible implementation, the recording task includes a voice collection task, the multimedia stream is an audio stream, the detection algorithm is used to detect the voice meaning in the audio stream, and the recording requirements are used to constrain the requirements met by the voice meaning in the audio stream.

[0026] In one possible embodiment, the configuration interface is also used to prompt the configuration of the presentation form of the recording suggestion; the acquisition module is also used to obtain second configuration information for the presentation form of the recording suggestion provided by the user in response to the user's operation on the configuration interface; the output module is used to: when the detection result does not meet the recording requirements, generate a recording suggestion in a first presentation form; according to the second configuration information, transform the recording suggestion in the first presentation form to obtain a recording suggestion in a second presentation form; and output the recording suggestion in the second presentation form.

[0027] In one possible implementation, the acquisition module is configured to: acquire a second configuration file input by a user on a configuration interface, the second configuration file including second configuration information; or, in response to a user's selection operation for identification of multiple candidate representation forms, obtain the second configuration information configured by the user.

[0028] In one possible implementation, the acquisition module is configured to: determine the recording task configured by the user in response to the user's selection operation on the configuration interface for multiple candidate recording task identifiers; determine the detection algorithm configured by the user in response to the user's selection operation on the configuration interface for multiple candidate detection algorithm identifiers; and determine the recording requirement configured by the user in response to the user's selection operation on the configuration interface for multiple candidate recording requirement identifiers.

[0029] In one possible implementation, multiple candidate detection algorithms and multiple candidate recording requirements are stored in a database, and the acquisition module is further used to: output an update interface, where the update interface is used to prompt the database to be updated; and update the candidate detection algorithms or candidate recording requirements in the database in response to the user's update operation on the database on the update interface.

[0030] In a possible implementation, the acquisition module is configured to acquire a first configuration file input by a user on a configuration interface, where the first configuration file includes a recording task, a detection algorithm, or a recording requirement.

[0031] In a possible implementation, the multimedia stream includes a video stream, and the output module is further used to: fuse the detection result or recording suggestion with the video stream to obtain a fused video stream; and output the fused video stream.

[0032] The multimedia stream processing device provided in the second aspect corresponds to the multimedia stream processing method provided in the first aspect. Therefore, the technical effects of the multimedia stream processing device provided in the second aspect can be referred to the relevant description of the technical effects of the above-mentioned first aspect or any implementation method of the first aspect, and will not be repeated here.

[0033] In a third aspect, the present application provides a computing device cluster, the computing device cluster includes at least one computing device, the at least one computing device includes at least one processor and at least one memory; the at least one memory is used to store instructions, and the at least one processor executes the instructions stored in the at least one memory, so that the computing device cluster executes the multimedia stream processing method in the above-mentioned first aspect or any possible implementation of the first aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The at least one computing device may also include a bus. The processor is connected to the memory via a bus. The memory may include a readable memory and a random access memory.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on at least one computing device, the at least one computing device executes the multimedia stream processing method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0035] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on at least one computing device, enables the at least one computing device to execute the multimedia stream processing method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0036] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0038] FIG1 is a schematic structural diagram of an exemplary multimedia stream processing system provided by the present application;

[0039] FIG2 is a flow chart of a multimedia stream processing method provided by the present application;

[0040] FIG3 is a schematic diagram of an exemplary configuration interface provided by this application;

[0041] FIG4 is a flow chart of a video stream processing method provided by the present application;

[0042] FIG5 is a flow chart of another video stream processing method provided by the present application;

[0043] FIG6 is a flow chart of another video stream processing method provided by the present application;

[0044] FIG7 is a flow chart of an audio stream processing method provided by the present application;

[0045] FIG8 is a schematic diagram of the structure of a computing device provided by the present application;

[0046] FIG9 is a schematic diagram of the structure of a computing device cluster provided in this application. DETAILED DESCRIPTION

[0047] The following will describe the solutions in the embodiments provided in this application in conjunction with the drawings in this application.

[0048] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate and are merely used to describe the manner in which objects with the same attributes are described in the embodiments of this application.

[0049] FIG1 is a schematic diagram illustrating the structure of an exemplary multimedia stream processing system 10. As shown in FIG1 , the multimedia stream processing system 10 includes a collection device 100, a processing device 200, and a client 300. The collection device 100, the processing device 200, and the client 300 can communicate with each other via a network. The client 300 can be an application running on a user's device, or a web browser provided by the multimedia stream processing system 10.

[0050] The acquisition device 100 is used to acquire multimedia streams, which can specifically be video streams or audio streams. For example, the acquisition device 100 can be an infrared camera, a depth camera, etc., so that by recording the captured object (such as a person), a corresponding video stream can be obtained; or the acquisition device 100 can be a microphone, etc., so that by recording the captured object, a corresponding audio stream can be obtained. Among them, whether to record or record the captured object can be determined based on the recording task configured by the user through the client 300.

[0051] In actual application, the collection device 100 can be integrated with the client 300. For example, the collection device 100 can be a terminal device with shooting or recording functions, and the client 300 can be deployed on the terminal device as an application; or, the collection device 100 and the client 300 can also be deployed separately. In this case, the collection device 100 can send the collected multimedia stream to the processing device 200 through the client 300, or the collection device 10 can also directly send the multimedia stream to the processing device 200.

[0052] The processing device 200 is used to process the received multimedia stream and output corresponding recording suggestions. The recording suggestions can be used to guide the captured object to perform audio or video recording, such as guiding the captured object to follow the recording requirements to complete video recording of facial expressions.

[0053] Specifically, as shown in Figure 1, processing device 200 may include an acquisition module 201, a detection module 202, and an output module 203. Acquisition module 201 may receive a configuration interface output by client 300, which prompts the user to configure the recording task, detection algorithm, and recording requirements. In response to the user's operations on the configuration interface, client 300 acquires the user-provided configuration information for the recording task, detection algorithm, and recording requirements. It also receives a multimedia stream associated with the recording task from client 300 and provides the multimedia stream to detection module 202, and the recording requirements to output module 203. Detection module 202 detects the multimedia stream based on the detection algorithm and obtains a detection result. For example, if the multimedia stream is a video stream recording facial expressions, the detection result may be, for example, "mouth open," "mouth closed," "eyes open," or "eyes closed," as identified by processing device 200. Detection module 202 then provides the detection result to output module 203. The output module 203 determines whether the detection result meets the recording requirements. If the detection result does not meet the recording requirements, a recording suggestion is generated for the captured subject. For example, if the output module 203 detects that the facial expression in the multimedia stream does not have "closed eyes" (i.e., does not meet the recording requirements), a recording suggestion can be generated to instruct the captured subject to make an "open eyes" expression. Finally, the output module 203 can output the recording suggestion to the client 300, so that the client 300 can present the recording suggestion to the user or the captured subject.

[0054] Since the processing device 200 can automatically process the recorded video stream / audio stream and output recording suggestions according to the recording requirements, the user does not need to determine the problems in the video / audio by repeatedly watching the video or repeatedly listening to the audio, but can directly re-record the video or audio based on the output recording suggestions, thereby effectively improving the efficiency of generating recorded video / audio that meets the user's expectations.

[0055] In actual application, during the process of recording video or audio, the acquisition device 100 can continuously provide multimedia streams to the processing device 200 (via the client 300). In this way, the processing device 200 can output recording suggestions for the current recording stage in real time according to the recording requirements. This allows the user to re-record only the current recording stage without having to re-record all recording stages, thereby further improving the efficiency of generating recorded video / audio.

[0056] Moreover, the processing device 200 can support users to independently configure recording tasks, detection algorithms and recording requirements, that is, users can independently select from a variety of candidate options corresponding to recording tasks, detection algorithms and recording requirements according to actual application needs, or users can customize recording tasks, detection algorithms and recording requirements, which can meet the differentiated needs of different users for different recording tasks, different detection algorithms and different recording requirements, thereby improving the flexibility of users in generating recorded videos / audio and enriching the application scenarios applicable to audio and video recording.

[0057] It is worth noting that the multimedia stream processing system 10 shown in Figure 1 is only an example and is not intended to limit the specific implementation of the multimedia stream processing system. For example, in the multimedia stream processing system 10 shown in Figure 1, the processing device 200 can be deployed in the cloud, such as a public cloud, a hybrid cloud, etc., to provide users with cloud services for processing multimedia streams, while the collection device 100 and the client can be deployed on the user side to facilitate the user to locally provide corresponding configuration information and generate corresponding multimedia streams. In other possible multimedia stream processing systems, only the collection device 100 and the processing device 200 may be included, and both the collection device 100 and the processing device 200 are deployed on the user side, so that the processing device 200 can provide users with localized multimedia stream processing services; at the same time, the processing device 200 can present an interactive interface to the user so that the user can interact with the processing device 200 through the interactive interface. Alternatively, in other possible multimedia stream processing systems, the multimedia stream processing system may include multiple collection devices and provide multiple clients to the outside world to achieve parallel processing of multiple multimedia streams. Alternatively, the multimedia stream processing system may include other types of devices to support the multimedia stream processing system with more functions. This application does not limit the specific architecture of the multimedia stream processing system.

[0058] In addition, the processing device 200 can be implemented by software or hardware. As an example, the implementation of the processing device 200 is described below.

[0059] The processing device 200, as an example of a software functional unit, may include code running on an instance, such as an agent running on an instance, and the instance may be at least one of a host, a virtual machine, a container, a thread, and a process. Furthermore, the above-mentioned instance may be one or more. For example, the processing device 200 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, and each AZ includes one data center or multiple data centers with close geographical locations. Generally, a region may include multiple AZs.

[0060] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0061] As an example of a hardware functional unit, the processing device 200 may include at least one computing device, such as a cloud server. Alternatively, the processing device 200 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), or any combination thereof.

[0062] When processing apparatus 200 includes multiple computing devices, these devices can be distributed in the same region or in different regions. The multiple computing devices included in processing apparatus 200 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in processing apparatus 200 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0063] It should be noted that the method steps executed by the various modules in the above-mentioned processing device 200 are only used as an exemplary description and are not intended to be limiting. For example, in other implementation examples, the acquisition module 201 in the processing device 200 can be used to execute the method steps executed by any module in the above-mentioned processing device 200, the detection module 202 can be used to execute the method steps executed by any module in the above-mentioned processing device 200, and the output module 203 can be used to execute the method steps executed by any module in the above-mentioned processing device 200. The steps that the acquisition module 201, the detection module 202, and the output module 203 are responsible for implementing can be specified as needed, and the full functions of the processing device 200 are realized by respectively implementing different method steps in the process of processing multimedia streams in the acquisition module 201, the detection module 202, and the output module 203.

[0064] To facilitate understanding, various non-limiting specific implementations of the multimedia stream processing process are described in detail below.

[0065] Referring to FIG2 , there is shown a flowchart illustrating a method for processing a multimedia stream according to an embodiment of the present application. This method can be applied to the multimedia stream processing system 10 shown in FIG1 , or can also be applied to other applicable multimedia stream processing systems. The following describes the process by which the modules in the processing device 200 process multimedia streams and output recording suggestions, using the multimedia stream processing system 10 shown in FIG1 as an example.

[0066] As shown in FIG2 , the method for processing multimedia streams may specifically include:

[0067] S201: The acquisition module 201 outputs a configuration interface to the client 300, where the configuration interface is used to prompt the configuration of the recording task, detection algorithm, and recording requirements.

[0068] S202: The client 300 sends the configuration information provided by the user to the acquisition module 201 in response to the user's operation on the configuration interface. The configuration information is used to configure the recording task, detection algorithm and recording requirements.

[0069] In this embodiment, the multimedia stream processing system 10 can support users to perform various recording tasks, such as video recording tasks for changes in body movements, video recording tasks for changes in facial expressions, video recording tasks for action redirection (redirecting real human body movements to characters in the virtual world), audio recording tasks for human or animal voices, etc., so that users can select corresponding tasks from these multiple recording tasks according to actual application needs.

[0070] Exemplarily, the acquisition module 201 can output a configuration interface as shown in FIG3 to the client 300 to prompt the user to configure the recording task on the configuration interface. As shown in FIG3, the configuration interface can include identifiers of multiple candidate recording tasks supported by the multimedia stream processing system 10. The candidate recording tasks can be, for example, body movement acquisition tasks, expression acquisition tasks, action redirection tasks, voice acquisition tasks, etc. In this way, the user can select the identifiers of the multiple candidate recording tasks on the configuration interface, so that the client 300 can provide the identifier of the candidate recording task selected by the user to the acquisition module 201 in response to the user's selection operation, so that the acquisition module 201 can determine the recording task configured by the user based on the identifier, that is, the candidate recording task indicated by the identifier selected by the user.

[0071] Furthermore, when the multiple candidate recording tasks presented on the configuration interface do not meet the user's recording requirements, such as when the user needs to record a video of an animal running (which does not conform to the aforementioned recording task), the user can also customize the recording task on the configuration interface. For example, the user can enter a configuration file on the configuration interface shown in FIG3 , which includes a customized recording task, etc. Exemplarily, the configuration file can be a JSON format file, or a configuration file in another format. The client 300 can then send the configuration file provided by the user to the acquisition module 201, so that the acquisition module 201 determines the recording task configured by the user based on the configuration file.

[0072] In addition to configuring recording tasks, users can also configure detection algorithms and recording requirements on the configuration interface presented by the client.

[0073] Among them, the detection algorithm is used to detect the recorded video stream or audio stream, such as detecting the body movements expressed by multiple frames in the video stream, or detecting the environmental noise, pronunciation emotions, etc. in the voice expressed in the audio stream.

[0074] Recording requirements are used to constrain the requirements that the content of the recorded video or audio stream must meet. For example, for recording a video of hand movements, the recording requirement may specifically require that the hand movement remain in a "still" state for at least 2 seconds.

[0075] Exemplarily, after the user completes the configuration of the recording task, the acquisition module 201 can further output the identifiers of one or more candidate detection algorithms and one or more candidate recording requirements corresponding to the recording task to the client 300 based on the recording task configured by the user, and prompt the user to configure the detection algorithm and recording requirements. As shown in Figure 3, the client 300 can also present the identifiers of multiple candidate detection algorithms and multiple recording requirements supported by the multimedia stream processing system 10 on the configuration interface. In this way, the user can continue to select the identifiers of multiple candidate detection algorithms and multiple candidate recording requirements on the configuration interface. Accordingly, the client 300 can provide the identifier of the detection algorithm task selected by the user and the identifier of the candidate recording task to the acquisition module 201 in response to the user's selection operation, so that the acquisition module 201 can determine the detection algorithm configured by the user (that is, the candidate detection algorithm indicated by the identifier selected by the user) and determine the recording requirements configured by the user (that is, the candidate recording requirements indicated by the identifier selected by the user) based on these identifiers.

[0076] Furthermore, if the multiple candidate detection algorithm identifiers and candidate recording requirement identifiers presented on the configuration interface do not meet user needs, the user can also customize the detection algorithm and recording requirements on the configuration interface. For example, the configuration file entered by the user on the configuration interface shown in FIG3 may include not only the name of the customized recording task, but also the user-defined detection algorithm and user-defined recording requirements. In this way, the acquisition module 201 can obtain the user-configured detection algorithm and recording requirements from the configuration file sent by the client 300.

[0077] It should be noted that the above-mentioned implementation method of obtaining configuration information by the obtaining module 201 is only for illustrative purposes and is not intended to be limiting. In actual applications, the obtaining module 201 may also obtain configuration information through other methods.

[0078] After receiving the configuration information, the acquisition module 201 may provide the identification of the detection algorithm configured by the user to the detection module 202 , and provide the identification of the recording requirement configured by the user to the output module 203 .

[0079] S202: The acquisition module 201 acquires the multimedia stream under the recording task configured by the user, where the multimedia stream includes a video stream or an audio stream, wherein the video stream is obtained by recording the video of the captured object, and the audio stream is obtained by recording the voice of the captured object.

[0080] After the user completes the configuration of the recording task, the acquisition module 201 may further acquire the video stream or audio stream of the recording task. For ease of description, the acquired video stream or audio stream is collectively referred to as multimedia stream below.

[0081] In one possible implementation, after receiving the configuration information sent by the client 300, the acquisition module 201 may send a request to the client 300 to obtain a multimedia stream. In response to the request, the client 300 uses the acquisition device 100 to record the video or audio of the captured object under the recording task configured by the user, thereby obtaining the multimedia stream under the recording task. The captured object refers to the object whose video or audio is recorded by the acquisition device 100, and can be a person, an animal, or other types of objects.

[0082] In actual application, during the process of recording the collected object, the client 300 can present an interactive interface to guide the collected object to perform video or audio recording. For example, when the recording task is specifically a limb movement collection task, the client 300 can present prompt information of the limb movement (such as the name of the limb movement, etc.) on the interactive interface. In this way, the collected object can make corresponding limb movements according to the prompt information; and, during the process of the collected object making limb movements, the collection device 100 can continue to shoot the collected object to record the behavior video of the collected object making limb movements. For another example, when the recording task is specifically a voice collection task, the client 300 can present a text on the interactive interface, and the collected object can pronounce according to the text content presented on the interactive interface; in this way, the collection device 100 can continue to collect the voice of the collected object and realize audio recording for the collected object.

[0083] Then, the acquisition device 100 can send the collected multimedia stream to the client 300, so that the client 300 can transmit the multimedia stream to the acquisition module 201 in the processing device 200, so that the acquisition module 201 obtains the multimedia stream. In other possible embodiments, the acquisition device 100 has wireless transmission capabilities, so that the acquisition device 100 can also remotely transmit the multimedia stream to the processing device 200 via the network.

[0084] In the process of recording the video or audio of the captured object, the acquisition device 100 continuously sends a multimedia stream to the processing device 200 (or to the processing device 200 through the client 300). Alternatively, after the acquisition device 100 completes the video or audio acquisition of the captured object within a period of time, it can send the captured audio or video to the processing device 200 (or send it to the processing device 200 through the client 300).

[0085] After acquiring the multimedia stream, the acquisition module 201 may provide the multimedia stream to the detection module 202 .

[0086] S203: The detection module 202 detects the multimedia stream according to the detection algorithm configured by the user and obtains a detection result.

[0087] In one possible implementation, when the configuration information acquired by the acquisition module 201 includes an identifier of a detection algorithm, the detection module 202 can access a database based on the identifier of the detection algorithm. The database pre-stores a plurality of detection algorithms (i.e., the candidate detection algorithms described above), so that the detection module 202 can access the detection algorithm configured by the user from the database. In addition, the database can also store a detection configuration corresponding to each detection algorithm. The detection configuration can be, for example, a value range of a plurality of meaning types. The value range of each meaning type can be used to constrain the value of the detection result of the same meaning. For example, when the movement distance of the hand position determined by the detection algorithm is within the value range of meaning type 1, it indicates that the hand position has not moved; and when the movement distance of the hand position determined by the detection algorithm is within the value range of meaning type 2, it indicates that the hand position has moved. Thus, the detection module 202 can detect the multimedia stream using the accessed detection algorithm and the detection configuration corresponding to the detection algorithm.

[0088] When the user configures the detection algorithm by uploading a configuration file, the detection module 202 can detect the multimedia stream according to the detection configuration in the configuration file by running the user-defined detection algorithm in the configuration file.

[0089] When detecting a multimedia stream, the detection module 202 may specifically detect the meaning expressed by the multimedia stream. Furthermore, the detection module 202 may also detect attribute information related to the meaning. This is illustrated below with reference to an example.

[0090] In the first example, the multimedia stream can specifically be a video stream obtained by recording the body movements of the subject being captured. In this case, the detection algorithm configured by the user is specifically an algorithm for detecting body movements in the video stream. The body movements can be, for example, hand movements such as "clenching a fist" or "clapping", or head movements such as "lowering the head", "raising the head", or "turning the head". Then, in the process of detecting the video stream using the detection algorithm, the detection module 202 can first identify the body in each frame of the video stream, and identify the body movements expressed by the multiple frames based on the differences between the body images in multiple consecutive frames; and the detection module 202 can also calculate the relevant attribute information of the body movement based on the multiple frames, such as the time it takes the subject to perform the body movement (which can be calculated based on the number of frames), the angle and amplitude of the body movement, and other information. Thus, the detection module 202 can generate a detection result including each body movement and the relevant attributes of each body movement.

[0091] In the second example, the multimedia stream can specifically be a video stream obtained by recording the facial expressions of the subject. In this case, the detection algorithm configured by the user is specifically an algorithm for detecting facial expressions in the video stream. Such facial expressions can include, for example, "mouth open," "mouth closed," "eyes open," "eyes closed," or "frown." Detection module 202, when using this detection algorithm to detect the video stream, can identify the facial expression included in each frame of the video stream. Furthermore, detection module 202 can also detect relevant attribute information of the facial expression in each frame, such as the angle and amplitude of the facial expression. Consequently, detection module 202 can generate detection results that include the facial expressions corresponding to multiple frames and the relevant attributes of each facial expression.

[0092] In the third example, the multimedia stream can specifically be a video stream obtained by recording the body movements of the captured subject. In this case, the detection algorithm configured by the user is specifically an algorithm for detecting movement problems that exist when body movements are redirected to a virtual character. The body movements can be, for example, "walking," "running," "bowing," "holding fists," and so on. Therefore, when using the detection algorithm to detect the video stream, the detection module 202 can first redirect the body movements of the captured subject in the video stream to the body movements of a virtual character (such as a digital human), thereby generating a video stream of the virtual character; then, the body movements in each frame of the video stream of the virtual character can be identified, and each frame can be checked for any problems with penetration (such as the mutual penetration and superposition of body A and body B, or the mutual penetration and superposition of body and the clothing of the virtual character), and for any abnormalities in the body movements expressed in multiple consecutive frames (such as sudden changes between the body movements in the previous frame and the next frame). Furthermore, the detection module 202 can further detect, based on the multiple frames, the duration of time during which the avatar's video stream is free of abnormalities, the duration of time during which the video stream contains model penetration issues, the duration of time during which abnormal body movements are present, the cause of the abnormality, and other related attribute information. Thus, the detection module 202 can generate a detection result including the avatar's video stream and related attributes of the video stream.

[0093] In the fourth example, the multimedia stream can specifically be an audio stream obtained by audio recording the voice of the collected object. At this time, the detection algorithm configured by the user is specifically an algorithm for detecting the voice meaning of the collected object. The voice meaning can be, for example, environmental noise, echo, phonemes, voice emotions, voice style, etc. present in the voice. Then, the detection module 202 can use the detection algorithm to detect voice meaning information such as environmental noise, echo, phonemes, voice emotions, or voice style in the audio stream. Furthermore, the detection module 202 can also detect attribute information related to the voice meaning information, such as further detecting the decibel and duration of the environmental noise, detecting the types and number of voice emotions, and other related attribute information. Thus, the detection module 202 can generate a detection result including voice meaning and related attributes of voice meaning.

[0094] It is understood that the above four implementations are merely exemplary and not limiting. For example, in other possible implementations, the detection module 202 may not need to detect the relevant attributes of the meaning expressed by the multimedia stream, or the meaning expressed by the multimedia stream detected by the detection module 202 may also be other types of meanings.

[0095] After generating the detection result, the detection module 202 may provide the detection result to the output module 203 .

[0096] S204: The output module 203 determines whether the detection result meets the recording requirements.

[0097] S205: When the detection result does not meet the recording requirement, the output module 203 outputs a recording suggestion for the captured object to the client 300.

[0098] In actual application scenarios, the content in the multimedia stream obtained by recording the video or audio of the captured subject may not meet the user's expectations. For example, when recording the subject's body movements, the recorded video stream may not include the user's expected "head turning" movement. Or, the video stream may include a "head turning" movement, but the angle of the "head turning" movement may not reach the user's expected 60 degrees.

[0099] Therefore, after obtaining the detection result, the output module 203 can further verify whether the detection result meets the recording requirements configured by the user. For example, the output module 203 can verify whether there is a "head turn" action in the video stream based on the detection result. If a "head turn" action is determined to be present, the output module 203 can further verify whether the head rotation angle corresponding to the "head turn" action in the video stream reaches the user's expected 60 degrees.

[0100] When it is determined that the detection result does not meet the recording requirements, the output module 203 can generate corresponding recording suggestions based on the detection result and the recording requirements. For example, when the head rotation angle corresponding to the "turn head" action in the video stream does not reach 60 degrees indicated by the recording requirements, the recording suggestion generated by the output module 203 can be "rotate the head to the left (or right)", or it can be "continue to rotate the head to the left (or right) until it reaches 60 degrees". For another example, when it is detected that the voice emotion of part of the audio in the audio stream does not meet the emotional requirements configured by the user (i.e., the recording requirements), the recording suggestion generated by the output module 203 can be "re-record the voice of the XXXX text".

[0101] Finally, the output module 203 can output the recording suggestion to the client 300, as shown in Figure 2, so that the client 300 can present the recording suggestion to the user. This way, the user can guide the subject to re-record the video or audio according to the presented recording suggestion, or re-record the video or audio clips with problems. Alternatively, the user can use the recording suggestion to promptly identify problems in the multimedia stream, allowing the user to decide whether to use the multimedia stream to generate a digital human.

[0102] In actual application, the output module 203 can also output the recording suggestion to the collection device 100, as shown in Figure 2, and the collection device 100 presents the recording suggestion to the collected object, such as outputting the recording suggestion content in audio form to the collected object, so that the collected object can re-execute the video or audio recording process according to the recording suggestion.

[0103] In a further possible implementation, the output module 203 may also transform the presentation form of the generated recording suggestions so that the recording suggestions output by the output module 203 can ultimately be presented to the user in a more user-friendly manner, thereby improving the user experience.

[0104] During specific implementation, the output module 203 generates a recording suggestion in a first form by default based on the detection results and the recording requirements, such as generating a recording suggestion in text form by default, or generating a recording suggestion in numerical form by default. Then, the output module 203 can transform the recording suggestion in the first form to obtain a recording suggestion in a second form, which can be, for example, a recording suggestion in the form of a picture, or a recording suggestion in the form of a voice. Then, the output module 203 outputs the recording suggestion in the second form, such as outputting the recording suggestion in the second form to the client 300 or the collection device 100. The second form can be any form that can improve the user experience, and can be set according to the needs of the actual application, and is not limited to this specific implementation.

[0105] Exemplarily, the second form of expression of the recording suggestion can be configured by the user. For example, before the output module 203 outputs the recording suggestion, the acquisition module 201 can output a configuration interface as shown in Figure 3 to the client 300. The configuration interface can also prompt the user to configure the form of expression of the recording suggestion, so that the user can perform configuration operations on the form of expression of the recording suggestion on the client 300. For example, the user can select identifiers for multiple candidate forms of expression on the configuration interface, so that the acquisition module 201 can determine the second form of expression configured by the user based on the identifier selected by the user; or, the user can enter a second configuration file on the configuration interface, and the second configuration file includes second configuration information for configuring the form of expression. In this way, the acquisition module 201 can provide the second configuration information provided by the user to the output module 203, so that the output module 203 can transform the recording suggestion of the first form of expression into the recording suggestion of the second form of expression configured by the user. In this way, not only can the diversity of the recording suggestions output by the processing device 200 be enriched, but also the user's selectivity for the form of expression of the recording suggestion can be improved, thereby further improving the user experience.

[0106] In other possible implementation examples, the second form of the recording suggestion may also be pre-configured in the processing device 200 by a technician, so that the output module 203 can output the recording suggestion in a fixed form, which is not limited in this embodiment.

[0107] Furthermore, in addition to outputting recording suggestions, the output module 203 can also output a processed multimedia stream, as shown in Figure 2. For example, the acquisition module 201 can provide the multimedia stream sent by the client 300 to the output module 203, so that the output module 203 can fuse the recording suggestions or the detection results generated by the detection module 202 with the multimedia stream and output the fused multimedia stream to the client 300. Taking the multimedia stream as a video stream as an example, the output module 203 can add the detection results to multiple frames in the video stream, such as marking the body movements / facial expressions / locations where the penetration occurred detected by the detection module 202 in the multiple frames included in the video stream, and then the output module 203 outputs a new video stream fused with the detection results. In this way, the user can view the detection status of the processing device 200 for the content in the video stream. Alternatively, the output module 203 can add the recording suggestions to multiple frames in the video stream, such as adding the recording suggestions to the multiple frames included in the video stream that suggest re-recording. In this way, the user can view the video portion that needs to be re-recorded in the video stream, so as to subsequently guide the captured subject to re-record that portion of the video.

[0108] In addition, the user can also update the candidate detection algorithms and candidate recording requirements stored in the database. In specific implementation, the acquisition module 201 can also output an update interface to the client 300, so that the client 300 presents the update interface to the user and prompts the user to update the database storing the candidate detection algorithms and candidate recording requirements. In this way, the user can perform update operations on the database on the update interface, such as providing a new detection algorithm or a new recording requirement to the processing device 200 through the client 300, or adjusting the existing candidate detection algorithm or candidate recording requirement. Accordingly, the acquisition module 201 can update the candidate detection algorithms and candidate recording requirements in the database in response to the selection operation performed by the user on the update interface, such as adding, deleting or modifying the candidate detection algorithms and candidate recording requirements in the database.

[0109] The method steps and the order in which they are executed shown in FIG2 are merely exemplary implementations and are not intended to be limiting. For example, in other possible embodiments, step S201 may be split into multiple method steps, such as a user configuring the recording task, detection algorithm, and recording requirements separately through different configuration interfaces. In this case, the step in which the acquisition module 201 acquires the detection algorithm and recording requirements may be placed after the step in which the acquisition module 201 acquires the multimedia stream.

[0110] Since the processing device 200 can automatically process the recorded video stream / audio stream and output recording suggestions according to the recording requirements, the user does not need to repeatedly watch the video or listen to the audio to determine the problem in the video / audio, but can directly re-record the video or audio based on the output recording suggestions, thereby effectively improving the efficiency of generating recorded video / audio that meets the user's expectations. Moreover, the processing device 200 can support the user to independently configure the recording task, detection algorithm and recording requirements, that is, the user can independently select from a variety of candidate options corresponding to the recording task, detection algorithm and recording requirements (and expression forms) according to the needs of the actual application, or the user can customize the recording task, detection algorithm and recording requirements (and expression forms), which can meet the differentiated needs of different users for different recording tasks, different detection algorithms, different recording requirements (and different expression forms), thereby improving the flexibility of users in generating recorded video / audio and enriching the application scenarios applicable to audio and video recording.

[0111] To facilitate further understanding of the technical solutions of the embodiments of the present application, the process of processing multimedia streams is introduced below in conjunction with a specific recording task.

[0112] Referring to FIG4 , a flow chart of a method for processing a video stream is shown. In the method embodiment shown in FIG4 , the recording task may specifically be a task of recording a video of a subject's body movements, where the body movements are hand movements, such as a "fist clenching" movement or a "hand-still" movement. As shown in FIG4 , the method may specifically include:

[0113] S401: The client 300 presents a configuration interface, which is used to prompt the user to configure the recording task, detection algorithm, recording requirements and presentation format.

[0114] Among them, the configuration interface presented by the client 300 can be provided by the acquisition module 201. The configuration interface can present identifications of multiple candidate items for recording tasks, detection algorithms, recording requirements and expression forms, so that the user can complete the corresponding configuration by selecting the identifications of multiple candidate items.

[0115] S402: Based on the user's configuration operation, the client 300 sends the configuration information to the acquisition module 201. The configuration information includes the identifier of the hand motion acquisition task, the identifier of the detection algorithm for detecting hand motions, the identifier of the recording requirements, and the identifier of the expression form. The client 300 provides the identifier of the detection algorithm to the detection module 202, and provides the identifier of the recording requirements and the identifier of the expression form to the output module 203.

[0116] S403: The acquisition module 201 instructs the client 300 to provide a video stream of the hand movements of the object to be captured according to the identifier of the hand movement capture task in the received configuration information.

[0117] S404: The client 300 instructs the collection device 100 to provide a recorded video of the hand movements of the collected object.

[0118] S405: The acquisition device 100 records the hand movements of the object being acquired, obtains a corresponding video stream, and provides the recorded video stream to the client 300.

[0119] During the video recording process, the client 300 may present corresponding prompt information to the subject, so that the subject makes corresponding hand movements according to the prompt information. Alternatively, the subject may make corresponding hand movements under the guidance of the user.

[0120] S406 : The client 300 forwards the video stream to the acquisition module 201 , and the acquisition module 201 provides the video stream to the detection module 202 .

[0121] S407: The detection module 202 accesses the detection algorithm configured by the user and the detection configuration corresponding to the detection algorithm from the database according to the identification of the detection algorithm.

[0122] S408 : The detection module 202 detects the hand movements in the video stream using a detection algorithm and a detection configuration, obtains a detection result, and provides the detection result to the output module 203 .

[0123] Exemplarily, the detection result includes multiple hand motions in the video stream and the duration of each hand motion.

[0124] S409: The output module 203 determines whether the detection result meets the recording requirements.

[0125] Exemplarily, the recording requirements specifically constrain the duration of each hand movement of the captured subject to a preset duration. The output module 203 can then determine each hand movement performed by the captured subject based on the detection results, and for each hand movement, the output module 203 further determines whether the duration of the hand movement has reached the preset duration. If so, the hand movement performed by the captured subject during the video recording meets the requirements; if not, the hand movement performed by the captured subject during the video recording does not meet the requirements. For example, if the captured subject's hand movement is a "fist clenching" movement, and the user-configured recording requirements specify that the duration of the "fist clenching" movement must be no longer than 5 seconds and no shorter than 2 seconds, then if the output module 203 determines that the duration of the "fist clenching" movement is less than 2 seconds or greater than 5 seconds, the hand movement is determined to not meet the recording requirements. If the output module 203 determines that the duration of the "fist clenching" movement is between 2 seconds and 5 seconds, the hand movement is determined to meet the recording requirements. For another example, the recording requirement is that the hand movement of the collected object remains in the "still state" for no less than 2 seconds before changing from the "still state" to the "manual state". After detecting the change in the hand movement of the collected object, the output module 203 can detect whether the duration of the hand movement of the collected object in the "still state" before the change is less than 2 seconds; if it is less, the output module 203 determines that the duration of the hand movement of the collected object remaining in the "still state" does not meet the recording requirement; if it is not less, the output module 203 determines that the hand movement of the collected object meets the recording requirement.

[0126] S410: When the detection result does not meet the recording requirement, the output module 203 generates a recording suggestion for the first representation of the captured object.

[0127] For example, when the duration of the "fist clenching" action is less than 2 seconds or greater than 5 seconds, the output module 203 can generate a recording suggestion including a "fist clenching" action identifier and a duration of 2 seconds to 5 seconds, so as to instruct the video of the "fist clenching" action of the captured object to be re-recorded, and the re-recorded "fist clenching" action is an action with a duration within the range of 2 seconds to 5 seconds.

[0128] The output module 203 generates a recording suggestion in a first expression form by default. The first expression form may be, for example, a text-type recording suggestion, such as the recording suggestion specifically including text-type information such as “make a fist” and “2 seconds to 5 seconds”.

[0129] In this embodiment, the output module 203 can output the recording suggestion in the form desired by the user. Specifically, the method shown in FIG4 can further include the following steps.

[0130] S411: The output module 203 transforms the recording suggestion in the first representation into the recording suggestion in the second representation according to the representation configured by the user.

[0131] In this embodiment, the second form of expression can be a form of expression that can improve the friendliness of human-computer interaction, such as a picture, video, voice, etc. For example, when the recording suggestion of the first form of expression is specifically "make a fist" or "2 seconds to 5 seconds", the recording suggestion of the second form of expression generated by the output module 203 can be specifically an audio corresponding to "please make a fist again, and the fist action should be about 3 seconds", or it can be a picture / video including the fist action and the duration. In this way, the user / collected object can obtain the recording suggestion in a more convenient or more preferred way, thereby improving the interactivity of human-computer interaction.

[0132] S412: The output module 203 outputs the recording suggestion in the second representation format to the client 300.

[0133] S413 : The output module 203 outputs the processed video stream to the client 300 , where the video stream includes the markers of each hand movement detected by the detection module 202 .

[0134] In this way, the captured object can reuse the capture device 100 to perform the recording process for hand movements that do not meet the recording requirements based on the recording suggestions and video stream output by the processing device 200. For the re-recorded video stream, the above-mentioned step S405 and subsequent steps can be re-executed until the hand movements of the captured object in the recorded video stream meet the recording requirements configured by the user. Among them, the re-recorded video stream can be a video stream obtained by re-recording only the hand movements that do not meet the recording requirements, so that the captured object does not need to re-record all hand movements, thereby improving the overall hand movement recording efficiency.

[0135] It is worth noting that the method embodiment shown in Figure 4 is only an exemplary description. In other possible embodiments, the output module 203 can also output the recording suggestion to the acquisition device 100, so that the collected object can directly determine the hand movements that do not meet the recording requirements on the collected object 100, so that the collected object can use the acquisition device 100 to perform the process of re-recording the hand movements.

[0136] In addition, the specific execution method of each step in the method embodiment shown in Figure 4 can be found in the relevant description of the embodiment shown in Figure 2 above, and will not be repeated here.

[0137] Referring to FIG5 , a flow chart of another method for processing a video stream is shown. In the method embodiment shown in FIG5 , the recording task may specifically be a task of recording a video of the facial expression of the subject being captured. The facial expression may be, for example, an expression such as "open mouth" or "closed eyes." As shown in FIG5 , the method may specifically include:

[0138] S501: The client 300 presents a configuration interface, which is used to prompt the user to configure the recording task, detection algorithm, recording requirements and presentation format.

[0139] S502: Based on the user's configuration operation, the client 300 sends configuration information to the acquisition module 201. The configuration information includes the identifier of the facial expression acquisition task, the identifier of the detection algorithm for detecting facial expressions, the identifier of the recording requirements, and the identifier of the expression form. The client 300 provides the identifier of the detection algorithm to the detection module 202, and provides the identifier of the recording requirements and the identifier of the expression form to the output module 203.

[0140] S503: The acquisition module 201 instructs the client 300 to provide a video stream of the facial expression of the object to be collected according to the identifier of the facial expression collection task in the received configuration information.

[0141] S504: The client 300 instructs the collection device 100 to provide a recorded video of the facial expression of the collected object.

[0142] S505: The collection device 100 records the facial expression of the collected object to obtain a corresponding video stream, and provides the recorded video stream to the client 300.

[0143] S506 : The client 300 forwards the video stream to the acquisition module 201 , and the acquisition module 201 provides the video stream to the detection module 202 .

[0144] S507: The detection module 202 accesses the detection algorithm configured by the user and the detection configuration corresponding to the detection algorithm from the database according to the identification of the detection algorithm.

[0145] S508 : The detection module 202 detects facial expressions in the video stream using a detection algorithm and a detection configuration, obtains a detection result, and provides the detection result to the output module 203 .

[0146] Exemplarily, the detection result includes multiple facial expressions in the video stream and the movement amplitude of each facial expression.

[0147] S509: The output module 203 determines whether the detection result meets the recording requirements.

[0148] Exemplarily, the recording requirements specifically restrict the amplitude of each facial expression of the captured subject to a value greater than a preset amplitude. Output module 203 can then determine each facial expression made by the captured subject based on the detection results, and for each facial expression, further determine whether the facial expression is greater than the preset amplitude. If so, the facial expression made by the captured subject during video recording meets the requirements; if not, the facial expression made by the captured subject during video recording does not meet the requirements. For example, if the facial expression of the captured subject is an "open mouth" expression, and the user-configured recording requirements indicate that the amplitude of the mouth opening in the "open mouth" expression is greater than a threshold value of 1, then if the output module 203 determines that the amplitude of the mouth opening in the "open mouth" expression is less than or equal to the threshold value of 1, the facial expression is determined to not meet the recording requirements. However, if the output module 203 determines that the amplitude of the mouth opening in the "open mouth" expression is greater than the threshold value of 1, the facial expression is determined to meet the recording requirements.

[0149] S510: When the detection result does not meet the recording requirement, the output module 203 generates a recording suggestion for the first representation of the captured object.

[0150] For example, when the opening amplitude of the mouth in the "open mouth" expression is less than the threshold value 1, the output module 203 can generate a recording suggestion including an "open mouth" expression identifier and an increase in the opening amplitude of the mouth, so as to instruct the video of the "open mouth" expression of the captured object to be re-recorded, and the re-recorded "open mouth" expression is specifically an expression with the mouth opened with a larger amplitude.

[0151] The output module 203 generates a recording suggestion in a first form by default. The first form may be, for example, a text-type recording suggestion, such as a recording suggestion specifically including text-type information such as "open your mouth" or "please open your mouth wider".

[0152] S511: The output module 203 transforms the recording suggestion in the first representation into a recording suggestion in the second representation according to the representation configured by the user.

[0153] In this embodiment, the second form of expression can be a form of expression that can improve the friendliness of human-computer interaction, such as a picture, video, voice, etc. For example, when the recording suggestion of the first form of expression is specifically "open your mouth" or "please open your mouth wider", the recording suggestion of the second form of expression generated by the output module 203 can be specifically the audio corresponding to "please make the mouth-opening expression again, open your mouth wider", or it can be a picture / video including the mouth-opening expression and the increased mouth-opening amplitude. In this way, the user / collected object can obtain the recording suggestion in a more convenient or more preferred way, thereby improving the interactivity of human-computer interaction.

[0154] S512: The output module 203 outputs the recording suggestion in the second representation format to the client 300.

[0155] S513 : The output module 203 outputs the processed video stream to the client 300 . The video stream includes the markers of each facial expression detected by the detection module 202 .

[0156] For the specific execution manner of each step in the method embodiment shown in FIG5 , please refer to the relevant description of the embodiment shown in FIG2 above, and will not be repeated here.

[0157] Referring to FIG6 , a flow chart of another method for processing a video stream is shown. In the method embodiment shown in FIG6 , the recording task may be specifically a task of redirecting the captured subject's body movements to a virtual character. As shown in FIG6 , the method may specifically include:

[0158] S601: The client 300 presents a configuration interface, which is used to prompt the user to configure the recording task, detection algorithm, recording requirements and presentation format.

[0159] S602: Based on the user's configuration operation, the client 300 sends the configuration information to the acquisition module 201. The configuration information includes the identifier of the action redirection task, the identifier of the detection algorithm for detecting action problems during the redirection process, the identifier of the recording requirements, and the identifier of the expression form. The identifier of the detection algorithm is provided to the detection module 202, and the identifier of the recording requirements and the identifier of the expression form are provided to the output module 203.

[0160] S603: The acquisition module 201 instructs the client 300 to provide a video stream of the body movements of the captured object according to the identifier of the movement redirection task in the received configuration information.

[0161] S604: The client 300 instructs the collection device 100 to provide a recorded video of the body movements of the collected object.

[0162] S605: The collection device 100 records the body movements of the collected object to obtain a corresponding video stream, and provides the recorded video stream to the client 300.

[0163] S606 : The client 300 forwards the video stream to the acquisition module 201 , and the acquisition module 201 provides the video stream to the detection module 202 .

[0164] S607: The detection module 202 accesses the detection algorithm configured by the user and the detection configuration corresponding to the detection algorithm from the database according to the identification of the detection algorithm.

[0165] S608: The detection module 202 redirects the body movements of the captured object in the video stream into the body movements of the virtual character, thereby generating a video stream of the virtual character.

[0166] The body movements of the virtual character in the video stream of the virtual character are consistent with the body movements of the captured object, thereby redirecting the body movements of the captured object to the virtual character.

[0167] S609 : The detection module 202 detects the motion problems in the video stream of the virtual character using the detection algorithm and the detection configuration, obtains the detection results, and provides the detection results to the output module 203 .

[0168] Exemplarily, the detection result includes detailed information such as the motion problems existing in the video stream of the virtual character (such as limb penetration, etc.) and the location of the motion problem in the video stream.

[0169] S610: The output module 203 determines whether the detection result meets the recording requirements.

[0170] For example, the recording requirements specifically constrain the virtual character's video stream to not have model penetration. Based on the detection results, the output module 203 can determine whether model penetration exists in the virtual character's video stream. If so, the video stream representing the virtual character does not meet the requirements; if not, the video stream representing the virtual character meets the requirements.

[0171] S611: When the detection result does not meet the recording requirement, the output module 203 generates a recording suggestion for the first representation of the captured object.

[0172] For example, when there is a problem of penetration between the arms and torso in the video stream of the virtual character, the output module 203 can generate a recording suggestion including an "arm" logo and raising the arm, so as to instruct the re-recording of the video of the captured object's limb movement of the "arm", and the re-recorded limb movement is specifically the action of "raising the arm".

[0173] Among them, the output module 203 generates a recording suggestion in the first form by default. The first form can be, for example, a text-type recording suggestion, such as the recording suggestion specifically including text-type information such as "There is penetration between the arm and the torso" and "Please raise your arms".

[0174] S612: The output module 203 transforms the recording suggestion in the first representation into a recording suggestion in the second representation according to the representation configured by the user.

[0175] In this embodiment, the second presentation format can be a format that enhances the user-friendly interaction between the user and the computer, such as images, videos, or audio. For example, when the recording suggestion in the first presentation format specifically includes "There is interference between the arm and the torso" or "Please raise your arms," ​​the recording suggestion in the second presentation format generated by the output module 203 can specifically include audio corresponding to "Please raise your arms." This allows the user or the subject being recorded to receive the recording suggestion in a more convenient or preferred manner, thereby enhancing the interactivity of the user-computer interaction.

[0176] S613: The output module 203 outputs the recording suggestion in the second representation format to the client 300.

[0177] S614: The output module 203 outputs the processed video stream of the virtual character to the client 300. The video stream includes the labels of each body movement detected by the detection module 202 and movement problems existing in some body movements.

[0178] For the specific execution manner of each step in the method embodiment shown in FIG6 , please refer to the relevant description of the embodiment shown in FIG2 above, and will not be repeated here.

[0179] Referring to FIG7 , a flow chart of a method for processing an audio stream is shown. In the method embodiment shown in FIG7 , the recording task may specifically be a task of recording the audio of the voice of the collected object. As shown in FIG7 , the method may specifically include:

[0180] S701: The client 300 presents a configuration interface, which is used to prompt the user to configure the recording task, detection algorithm, recording requirements and presentation format.

[0181] S702: Based on the user's configuration operation, the client 300 sends the configuration information to the acquisition module 201. The configuration information includes the identifier of the voice collection task, the identifier of the detection algorithm for detecting action problems during the redirection process, the identifier of the recording requirements, and the identifier of the expression form. The identifier of the detection algorithm is provided to the detection module 202, and the identifier of the recording requirements and the identifier of the expression form are provided to the output module 203.

[0182] In actual application, the user's configuration of recording requirements can also be expressed as a configuration of application scenarios for voice collection. At this time, for different application scenarios configured by the user, the collection device 100 can use different recording texts to record the voice of the collected object.

[0183] S703: The acquisition module 201 instructs the client 300 to provide an audio stream of the voice of the collected object according to the identifier of the voice collection task in the received configuration information.

[0184] S704: The client 300 instructs the collection device 100 to provide recorded audio of the voice of the collected object.

[0185] S705: The collection device 100 records the voice of the collected object to obtain a corresponding audio stream, and provides the recorded audio stream to the client 300.

[0186] Among them, the collected device can output voice based on a specific recording text, and the recording text can be determined according to the recording requirements configured by the user (ie, the application scenario).

[0187] S706 : The client 300 forwards the audio stream to the acquisition module 201 , and the acquisition module 201 provides the audio stream to the detection module 202 .

[0188] S707: The detection module 202 obtains the detection algorithm configured by the user and the detection configuration corresponding to the detection algorithm from the database according to the identification of the detection algorithm.

[0189] S708 : The detection module 202 detects the voice of the collected object in the audio stream, obtains a detection result, and provides the detection result to the output module 203 .

[0190] For example, the detection result includes the speech meaning included in the audio stream. The speech meaning may include, for example, ambient noise, echo, pronunciation of the subject, speech emotion, and speech style. Furthermore, the detection result may also include attribute information corresponding to the speech meaning, such as the decibel level of the ambient noise.

[0191] S709: The output module 203 determines whether the detection result meets the recording requirements.

[0192] Illustratively, the recording requirements are specifically used to constrain the meaning of speech in the audio stream.

[0193] When the speech meaning is specifically environmental noise, the output module 203 may specifically determine whether the decibel of the environmental noise in the audio stream is greater than a threshold value 1, that is, determine whether the recording environment of the collected object is quiet.

[0194] When the speech meaning is specifically an echo, the output module 203 may specifically determine whether the decibel of the echo in the audio stream is greater than a threshold of 2, that is, determine whether the collected object will generate an echo that affects the recording quality during recording.

[0195] When the speech meaning is specifically pronunciation, the output module 203 can specifically determine whether the pronunciation of the collected object conforms to the standard pronunciation corresponding to the recording text, that is, determine whether the pronunciation of the collected object has problems such as inaccurate pronunciation or unclear pronunciation.

[0196] When the speech meaning is specifically speech emotion, the output module 203 can specifically determine whether the speech emotion corresponding to the pronunciation of the collected object in the audio stream is a preset emotion, such as determining whether the pronunciation of the collected object includes the tone of the preset emotion.

[0197] When the speech meaning is specifically a speech style, the output module 203 may specifically determine whether the speech style corresponding to the pronunciation of the collected object in the audio stream is a preset style, such as determining whether the pronunciation of the collected object includes the tone of the preset style.

[0198] S710: When the detection result does not meet the recording requirement, the output module 203 generates a recording suggestion for the first representation of the captured object.

[0199] For example, when the pronunciation of the collected object in the audio stream lacks obvious expression of emotion / style, the output module 203 can generate a recording suggestion including an "emotion" identifier / "style" identifier to instruct the re-recording of the audio of the collected object for a specific emotion / specific style, and the emotion / style of the re-recorded audio pronunciation is the preset emotion / style.

[0200] The output module 203 generates the recording suggestion in a first expression form by default. The first expression form may be, for example, a text-type recording suggestion, such as a text-type message such as "Please record in XX style".

[0201] S711: The output module 203 transforms the recording suggestion in the first representation into a recording suggestion in the second representation according to the representation configured by the user.

[0202] In this embodiment, the second presentation format can be a format that enhances the user-friendly interaction between the user and the computer, such as images, videos, or voice. For example, if the recording suggestion in the first presentation format is a text message such as "Please record in the style of XX," the second presentation format generated by the output module 203 can be an audio recording corresponding to "Please record in the style of XX." This allows the user or subject to receive the recording suggestion in a more convenient or preferred manner, thereby enhancing the interactivity of the user-computer interaction.

[0203] S712: The output module 203 outputs the recording suggestion in the second representation format to the client 300.

[0204] S713 : The output module 203 outputs the processed audio stream to the client 300 . The audio stream includes the detection result detected by the detection module 202 .

[0205] For the specific execution manner of each step in the method embodiment shown in FIG7 , please refer to the relevant description of the embodiment shown in FIG2 above, and will not be repeated here.

[0206] In the embodiments shown in Figures 1 to 7 above, processing device 200 may be software configured on a computing device or computing device cluster. Running this software on the computing device or computing device cluster enables the computing device or computing device cluster to implement the functions of processing device 200. The following describes processing device 200 in detail from the perspective of hardware implementation.

[0207] Figure 8 shows a structural diagram of a computing device, on which the above-mentioned processing device 200 can be deployed. The computing device can be a computing device in a cloud environment (such as a server), or a computing device in an edge environment, etc., and can be specifically used to implement the functions of the processing device 200 in the embodiments shown in Figures 2 and 4 to 7 above.

[0208] As shown in Figure 8, the computing device 800 includes a processor 810, a memory 820, a communication interface 830 and a bus 840. The processor 810, the memory 820 and the communication interface 830 communicate with each other via the bus 840. The bus 840 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in Figure 8, but it does not mean that there is only one bus or one type of bus. The communication interface 830 is used to communicate with the outside, for example, to obtain a first video stream, send a second video stream, etc.

[0209] The processor 810 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits. The processor 810 may also be an integrated circuit chip with signal processing capabilities. During implementation, the functions of each device in the processing device 200 may be implemented by hardware integrated logic circuits or software instructions in the processor 810. The processor 810 may also be a general-purpose processor, a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The methods disclosed in the embodiments of this application may be directly implemented as a hardware decoding processor, or may be implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or the like. The storage medium is located in the memory 820 , and the processor 810 reads the information in the memory 820 and completes part or all of the functions of the processing device 200 in combination with its hardware.

[0210] The memory 820 may include a volatile memory, such as a random access memory (RAM), or a non-volatile memory, such as a read-only memory (ROM), a flash memory, a HDD, or an SSD.

[0211] The memory 820 stores executable codes, and the processor 810 executes the executable codes to perform the method performed by the aforementioned processing device 200 .

[0212] Specifically, when implementing the embodiments shown in Figures 2 and 4 to 7, and when the processing device 200 described in the embodiments shown in Figures 2 and 4 to 7 is implemented by software, the software or program code required to execute the functions of the processing device 200 in Figures 2 and 4 to 7 is stored in the memory 820, and the interaction between the processing device 200 and the acquisition device 100 is realized through the communication interface 830. The processor is used to execute the instructions in the memory 820 to implement the method executed by the processing device 200.

[0213] FIG9 is a schematic diagram illustrating the structure of a computing device cluster. The computing device cluster 90 shown in FIG9 includes multiple computing devices, and the processing apparatus 200 described above can be deployed in a distributed manner on multiple computing devices in the computing device cluster 90. As shown in FIG9 , the computing device cluster 90 includes multiple computing devices 900, each of which includes a memory 920, a processor 910, a communication interface 930, and a bus 940. The memory 920, the processor 910, and the communication interface 930 are communicatively connected to each other via the bus 940.

[0214] Processor 910 can be implemented as a CPU, GPU, ASIC, or one or more integrated circuits. Processor 910 can also be an integrated circuit chip with signal processing capabilities. During implementation, some functions of processing device 200 can be performed using hardware integrated logic circuits or software instructions within processor 910. Processor 910 can also be a DSP, FPGA, general-purpose processor, other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components, and can implement or execute some of the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software modules can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory 920. In each computing device 900, processor 910 reads information from memory 920 and, in conjunction with its hardware, can perform some of the functions of processing device 200.

[0215] The memory 920 may include ROM, RAM, static storage devices, dynamic storage devices, hard disks (e.g., SSDs, HDDs), etc. The memory 920 may store program codes, for example, part or all of the program codes for implementing the acquisition module 201, part or all of the program codes for implementing the detection module 202, part or all of the program codes for implementing the output module 203, etc. For each computing device 900, when the program codes stored in the memory 920 are executed by the processor 910, the processor 910 executes part of the method executed by the processing device 200 based on the communication interface 930. The memory 920 may also store data, for example, intermediate data or result data generated by the processor 910 during the execution process, for example, the above-mentioned detection results, recording suggestions, etc.

[0216] The communication interface 903 in each computing device 900 is used for external communication, such as interacting with other computing devices 900 .

[0217] The bus 940 may be a peripheral component interconnect standard bus or an extended industry standard architecture bus, etc. For ease of illustration, the bus 940 in each computing device 900 in FIG9 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0218] The plurality of computing devices 900 establish communication paths through a communication network to implement the functions of the processing apparatus 200. Any computing device may be a computing device (eg, a server) in a cloud environment or a computing device in an edge environment.

[0219] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on one or more computing devices, the one or more computing devices execute the method executed by the processing device 200 of the above embodiment.

[0220] In addition, embodiments of the present application further provide a computer program product. When executed by one or more computing devices, the one or more computing devices perform any of the aforementioned methods for processing multimedia streams. The computer program product may be a software installation package. When any of the aforementioned methods for processing multimedia streams is required, the computer program product may be downloaded and executed on a computer.

[0221] It should also be noted that the system embodiments described above are merely illustrative, wherein the devices described as separate components may or may not be physically separate, and the components shown as devices may or may not be physical devices, that is, they may be located in one place or distributed across multiple network devices. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the system embodiments provided in this application, the connection relationship between the devices indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0222] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0223] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0224] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

Claims

1. A method for processing multimedia streams, characterized in that: The method comprises: Output a configuration interface, wherein the configuration interface is used to prompt configuration of recording tasks, detection algorithms, and recording requirements; In response to an operation of a user on the configuration interface, obtaining first configuration information provided by the user, where the first configuration information is used to configure a recording task, a detection algorithm, and a recording requirement; Acquire a multimedia stream under the recording task, wherein the multimedia stream includes a video stream or an audio stream, wherein the video stream is obtained by recording a video of the object to be captured, and the audio stream is obtained by recording a voice of the object to be captured; According to the detection algorithm, the multimedia stream is detected to obtain a detection result; When the detection result does not meet the recording requirement, a recording suggestion for the captured object is output.

2. The method according to claim 1, characterized in that The recording requirement is used to constrain requirements to be met by video content in the video stream obtained by recording, or is used to constrain requirements to be met by audio content in the audio stream obtained by recording.

3. The method according to claim 1 or 2, characterized in that: The recording task includes a limb motion collection task, the multimedia stream is a video stream, the detection algorithm is used to detect limb motions in the video stream, and the recording requirements are used to constrain requirements satisfied by the limb motions in the video stream.

4. The method according to claim 1 or 2, characterized in that: The recording task includes an expression collection task, the multimedia stream is a video stream, the detection algorithm is used to detect facial expressions in the video stream, and the recording requirements are used to constrain requirements satisfied by facial expressions in the video stream.

5. The method according to claim 1 or 2, characterized in that: The recording task includes an action redirection task, the multimedia stream is a video stream, the detection algorithm is used to detect action problems existing in redirecting body movements in the video stream to a virtual character, and the recording requirements are used to constrain the requirements met by the body movements in the video stream redirected to the virtual character.

6. The method according to claim 1 or 2, characterized in that: The recording task includes a voice collection task, the multimedia stream is an audio stream, the detection algorithm is used to detect the voice meaning in the audio stream, and the recording requirement is used to constrain the requirements met by the voice meaning in the audio stream.

7. The method according to any one of claims 1 to 6, characterized in that: The configuration interface is also used to prompt the configuration of the presentation form of the recording suggestion, and the method further includes: In response to an operation of the user on the configuration interface, obtaining second configuration information of a presentation form of the recording suggestion provided by the user; When the detection result does not meet the recording requirement, outputting a recording suggestion for the captured object includes: When the detection result does not meet the recording requirement, generating a recording suggestion in a first expression form; According to the second configuration information, the recording suggestion in the first expression form is transformed to obtain the recording suggestion in the second expression form; Output a recording suggestion in the second expression form.

8. The method according to claim 7, characterized in that The acquiring, in response to the user's operation on the configuration interface, second configuration information for the presentation form of the recording suggestion provided by the user includes: Acquire a second configuration file input by the user on the configuration interface, where the second configuration file includes the second configuration information; Alternatively, in response to the user's selection operation on identification of a plurality of candidate representation forms, the second configuration information configured by the user is obtained.

9. The method according to any one of claims 1 to 8, characterized in that: The step of obtaining, in response to an operation of the user on the configuration interface, first configuration information provided by the user includes: In response to a selection operation by the user on the configuration interface for identifications of a plurality of candidate recording tasks, determining a recording task configured by the user; In response to the user's selection operation on the configuration interface for identifications of multiple candidate detection algorithms, determining the detection algorithm configured by the user; In response to the user's selection operation on the configuration interface for identification of multiple candidate recording requirements, the recording requirement configured by the user is determined.

10. The method according to claim 9, characterized in that The multiple candidate detection algorithms and the multiple candidate recording requirements are stored in a database, and the method further includes: Outputting an update interface, wherein the update interface is used to prompt to update the database; In response to a user's update operation on the database on the update interface, the candidate detection algorithm or the candidate recording requirement in the database is updated.

11. The method according to any one of claims 1 to 8, characterized in that: The step of obtaining, in response to an operation of the user on the configuration interface, first configuration information provided by the user includes: A first configuration file input by a user on the configuration interface is obtained, where the first configuration file includes the recording task, the detection algorithm, or the recording requirement.

12. The method according to any one of claims 1 to 11, characterized in that: The multimedia stream includes a video stream, and the method further includes: Merging the detection result or the recording suggestion with the video stream to obtain a fused video stream; The fused video stream is output.

13. A multimedia stream processing device, characterized in that: The device comprises: an acquisition module, configured to output a configuration interface, wherein the configuration interface is used to prompt the user to configure a recording task, a detection algorithm, and a recording requirement; in response to an operation of a user on the configuration interface, acquire first configuration information provided by the user, wherein the first configuration information is used to configure the recording task, the detection algorithm, and the recording requirement; acquire a multimedia stream under the recording task, wherein the multimedia stream includes a video stream or an audio stream, wherein the video stream is obtained by recording a video of a captured object, and the audio stream is obtained by recording a voice of a captured object; A detection module, used to detect the multimedia stream according to the detection algorithm to obtain a detection result; The output module is used to output a recording suggestion for the captured object when the detection result does not meet the recording requirement.

14. The device according to claim 13, characterized in that The recording requirement is used to constrain requirements to be met by video content in the video stream obtained by recording, or is used to constrain requirements to be met by audio content in the audio stream obtained by recording.

15. The device according to claim 13 or 14, characterized in that The recording task includes a limb motion collection task, the multimedia stream is a video stream, the detection algorithm is used to detect limb motions in the video stream, and the recording requirements are used to constrain requirements satisfied by the limb motions in the video stream.

16. The device according to claim 13 or 14, characterized in that The recording task includes an expression collection task, the multimedia stream is a video stream, the detection algorithm is used to detect facial expressions in the video stream, and the recording requirements are used to constrain requirements satisfied by facial expressions in the video stream.

17. The device according to claim 13 or 14, characterized in that The recording task includes an action redirection task, the multimedia stream is a video stream, the detection algorithm is used to detect action problems existing in redirecting body movements in the video stream to a virtual character, and the recording requirements are used to constrain the requirements met by the body movements in the video stream redirected to the virtual character.

18. The device according to claim 13 or 14, characterized in that The recording task includes a voice collection task, the multimedia stream is an audio stream, the detection algorithm is used to detect the voice meaning in the audio stream, and the recording requirement is used to constrain the requirements met by the voice meaning in the audio stream.

19. The device according to any one of claims 13 to 18, characterized in that The configuration interface is also used to prompt configuration of the presentation form of the recording suggestion; The acquisition module is further configured to acquire, in response to an operation of the user on the configuration interface, second configuration information for the expression form of the recording suggestion provided by the user; The output module is used for: When the detection result does not meet the recording requirement, generating a recording suggestion in a first expression form; According to the second configuration information, the recording suggestion in the first expression form is transformed to obtain the recording suggestion in the second expression form; Output a recording suggestion in the second expression form.

20. The device according to claim 19, characterized in that The acquisition module is used to: Acquire a second configuration file input by the user on the configuration interface, where the second configuration file includes the second configuration information; Alternatively, in response to the user's selection operation on identification of a plurality of candidate representation forms, the second configuration information configured by the user is obtained.

21. The device according to any one of claims 13 to 20, characterized in that The acquisition module is used to: In response to a selection operation by the user on the configuration interface for identifications of a plurality of candidate recording tasks, determining a recording task configured by the user; In response to the user's selection operation on the configuration interface for identifications of multiple candidate detection algorithms, determining the detection algorithm configured by the user; In response to the user's selection operation on the configuration interface for identification of multiple candidate recording requirements, the recording requirement configured by the user is determined.

22. The device according to claim 21, characterized in that The multiple candidate detection algorithms and the multiple candidate recording requirements are stored in a database, and the acquisition module is further used to: Outputting an update interface, wherein the update interface is used to prompt to update the database; In response to a user's update operation on the database on the update interface, the candidate detection algorithm or the candidate recording requirement in the database is updated.

23. The device according to any one of claims 13 to 20, characterized in that The acquisition module is used to acquire a first configuration file input by a user on the configuration interface, where the first configuration file includes the recording task, the detection algorithm, or the recording requirement.

24. The device according to any one of claims 13 to 23, characterized in that The multimedia stream includes a video stream, and the output module is further used for: Merging the detection result or the recording suggestion with the video stream to obtain a fused video stream; The fused video stream is output.

25. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the computing device cluster executes the method according to any one of claims 1 to 12.

26. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on at least one computing device, enable the at least one computing device to perform the method according to any one of claims 1 to 12.

Images