Live broadcast image generation method and device based on digital human, and electronic equipment

Abstract

The invention provides a live broadcast image generation method and device based on a digital human and electronic equipment, and relates to the technical field of computers, in particular to the technical fields of artificial intelligence, computer vision, augmented reality, natural language processing, large models, digital live broadcast and the like. According to the specific implementation scheme, the method comprises the steps of obtaining at least one piece of first monitoring information of people flow information of a target scene, operation data of interaction equipment arranged in the target scene, a scene environment image and environment quality information; generating a target prompt word according to the first monitoring information; inputting the target cue word into a language model to obtain explanation content output by the language model; generating a first live broadcast image of the digital person according to the explanation content; and fusing the scene environment image and the first live broadcast image of the digital person to obtain a second live broadcast image of the digital person live broadcast scene.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to artificial intelligence, computer vision, augmented reality, natural language processing, large models, digital live broadcast and other technical fields, and can be applied to video and live broadcast scenarios, in particular to a live broadcast image generation method, device and electronic equipment based on digital humans. Background Art

[0002] Currently, slow livestreaming projects are being deployed on a large scale in diverse settings, including scenic spots, urban landmarks, and ecological conservation areas. These projects rely on fixed cameras or dynamic tracking lenses to continuously transmit real-time dynamic information within the scene. However, these livestreaming models generally suffer from the dilemma of "one-way visual output and shallow content reach." While they achieve the visual transmission of spatial information, the lack of interactive design, insufficient content depth, and personalized services results in short user stays and low engagement, making it difficult to generate sustained communication value.

[0003] As a typical carrier of virtual-reality symbiosis technology, the core value of digital humans lies in using virtual images to transcend the spatial and temporal constraints and resource limitations of the physical world. In live streaming, how to use technological integration to enable digital humans to become content enhancers and interactive hubs is the key to promoting the transition from one-way communication to two-way immersion. Summary of the Invention

[0004] The present disclosure provides a method, device and electronic device for generating a live broadcast image based on a digital human.

[0005] According to one aspect of the present disclosure, a method for generating a live broadcast image based on a digital human is provided, the method comprising: obtaining at least one first monitoring information selected from crowd flow information of a target scene, operation data of an interactive device set in the target scene, a scene environment image, and environmental quality information; generating a target prompt word based on the first monitoring information; inputting the target prompt word into a language model to obtain explanation content output by the language model; generating a first live broadcast image of the digital human based on the explanation content; and fusing the scene environment image with the first live broadcast image of the digital human to obtain a second live broadcast image of the digital human live broadcast scene.

[0006] According to another aspect of the present disclosure, a device for generating a live broadcast image based on a digital human is provided, the device comprising: an acquisition module for acquiring at least one first monitoring information selected from crowd flow information of a target scene, operation data of interactive devices arranged in the target scene, a scene environment image, and environmental quality information; a prompt module for generating a target prompt word based on the first monitoring information; a processing module for inputting the target prompt word into a language model to obtain explanation content output by the language model; a generation module for generating a first live broadcast image of the digital human based on the explanation content; and a fusion module for fusing the scene environment image with the first live broadcast image of the digital human to obtain a second live broadcast image of the digital human live broadcast scene.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the live broadcast image generation method based on digital humans proposed above in the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the live broadcast image generation method based on digital humans proposed above.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned method for generating a live broadcast image based on a digital human proposed in the present disclosure.

[0010] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0012] Figure 1 This is a flow chart of a method for generating a live broadcast image based on a digital human according to an embodiment of the present disclosure;

[0013] Figure 2 is a flow chart of a method for generating a live broadcast image based on a digital human according to another embodiment of the present disclosure;

[0014] Figure 3is a flow chart of a method for generating a live broadcast image based on a digital human according to another embodiment of the present disclosure;

[0015] Figure 4 is a flow chart of a method for generating a live broadcast image based on a digital human according to another embodiment of the present disclosure;

[0016] Figure 5 This is a schematic structural diagram of the digital human-based live broadcast image generation system provided by the present disclosure;

[0017] Figure 6 is a structural diagram of a live broadcast image generation device based on a digital human according to another embodiment of the present disclosure;

[0018] Figure 7 It is a block diagram of an electronic device used to implement the method for generating a live broadcast image based on a digital human according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, comply with relevant laws and regulations, and do not violate public order and good morals.

[0021] At present, slow live broadcasts mostly rely on fixed cameras for one-way real-scene recording, and the essence of their content presentation is still a "landscape window display". Due to the lack of dynamic interactive mechanisms and personalized service design, the audience stickiness is insufficient, making it difficult to form continuous traffic retention.

[0022] Although digital human technology has been implemented in some fields, the following problems are common:

[0023] 1. Single interaction dimension: Most digital humans only support voice or text interaction and lack multimodal collaborative response;

[0024] 2. Weak scene adaptability: Unable to generate relevant content in real time based on live broadcast images.

[0025] In response to at least one of the above problems, the present disclosure proposes a method, device and electronic device for generating a live broadcast image based on a digital human.

[0026] The following describes a method, device, and electronic device for generating a live broadcast image based on a digital human according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0027] It should be noted that the execution subject of the live broadcast image generation method based on digital humans in the embodiment of the present disclosure is a live broadcast image generation device based on digital humans. The live broadcast image generation device based on digital humans can be implemented by software and / or hardware and can be configured in electronic devices. The electronic devices may include but are not limited to terminals, servers, etc.

[0028] Figure 1 The present invention is a flowchart of a method for generating a live broadcast image based on a digital human according to an embodiment of the present invention.

[0029] like Figure 1 As shown, the method for generating a live broadcast image based on a digital human may include the following steps:

[0030] Step 101: Obtain at least one first monitoring information selected from among crowd flow information of a target scene, operation data of an interactive device set in the target scene, a scene environment image, and environment quality information.

[0031] The target scenes may include but are not limited to: scenic spots, studios, hospitals, classrooms, etc.

[0032] The crowd flow information may include but is not limited to: population size, population density, etc.

[0033] The interactive device may be, for example, a smart guide screen, an interactive advertising screen, or the like in an IoT (Internet of Things) device.

[0034] Among them, operation data refers to the behavior and interaction data generated by the user in the target scenario during the interaction with the interactive device.

[0035] The scene environment image is a real-time presentation and dynamic capture of the target scene, for example, in the form of a continuous video stream.

[0036] Among them, environmental quality information may include but is not limited to: meteorological information (such as temperature, air humidity, weather type, rainfall type, rainfall intensity, wind speed, light intensity, etc.), noise level, season, day and night information (including day, night, etc.), etc.

[0037] In an embodiment of the present disclosure, at least one first monitoring information selected from the following may be obtained: crowd flow information of a target scene, operation data of an interactive device set in the target scene, scene environment images, and environmental quality information. In one example, assuming that the target scene is a scenic spot, a 360° panoramic camera deployed at a key point in the scenic spot (such as an entrance, an observation deck, etc.) may be used to obtain the scene environment images, wherein the 360° panoramic camera may, for example, support 8K resolution, HDR (High Dynamic Range) dynamic exposure, and real-time streaming via RTMP (Real-Time Messaging Protocol); intelligent monitoring of the crowd density or population of a scenic spot or attraction may be achieved by setting up a millimeter-wave radar and camera in a coordinated manner with the scenic spot; scenic spot temperature, air humidity, rainfall information, wind speed, etc. in environmental quality information may be obtained through a weather station, and the ambient noise decibel of the scenic spot may be identified through an acoustic sensor set in the scenic spot; the number of visitors entering the park may be obtained in real time through an IoT device gate set in the scenic spot, and the operation data of tourists may be obtained through an interactive device smart guide screen set under the scenic spot, such as click query information, etc.

[0038] Step 102: Generate a target prompt word according to the first monitoring information.

[0039] In the embodiment of the present disclosure, a target prompt word may be generated based on the first monitoring information.

[0040] Step 103: Input the target prompt word into the language model to obtain the explanation content output by the language model.

[0041] The language model may be a large model having the function of generating explanation content based on prompt words.

[0042] In the embodiment of the present disclosure, the target prompt word may be input into a language model to obtain explanation content based on the output of the language model.

[0043] Step 104: Generate a first live broadcast image of the digital human according to the explanation content.

[0044] The first live broadcast image of the digital human may be in the form of a video stream.

[0045] In the embodiment of the present disclosure, a first live broadcast image of the digital human may be generated according to the explanation content.

[0046] Step 105: Fusing the scene environment image and the first live broadcast image of the digital human to obtain a second live broadcast image of the digital human live broadcast scene.

[0047] Among them, the second live broadcast image can be a picture-in-picture style, a virtual symbiosis style, etc., and the present disclosure does not impose any restrictions on this and can be set according to actual needs.

[0048] In one example, OpenCV image processing technology can be used to dynamically fuse and render the scene environment image and the first live broadcast image of the digital human in real time to obtain a second live broadcast image of the digital human live broadcast scene.

[0049] The method for generating live broadcast images based on a digital human in the disclosed embodiment obtains at least one first monitoring information selected from the following: crowd flow information of a target scene, operation data of interactive devices set in the target scene, scene environment images, and environmental quality information; generates a target prompt word based on the first monitoring information; inputs the target prompt word into a language model to obtain explanation content output by the language model; generates a first live broadcast image of the digital human based on the explanation content; and fuses the scene environment image and the first live broadcast image of the digital human to obtain a second live broadcast image of the digital human live broadcast scene. Thus, the explanation content generated by the language model based on multimodal data can dynamically adapt to the needs of different user groups, thereby providing in-depth explanations of the scene based on the digital human; thus, the scene environment image and the live broadcast image of the digital human are integrated to achieve an organic combination of intuitive display and in-depth explanation, avoiding the drawbacks of traditional live broadcasts where simple visual display lacks connotation or simple explanation is divorced from the actual scene. This can effectively improve the quality of the live broadcast image and enhance the richness and real-time nature of the image content.

[0050] In order to clearly illustrate how the target prompt word is generated according to the first monitoring information in the above embodiment of the present disclosure, the present disclosure also proposes a live broadcast image generation method based on digital humans.

[0051] Figure 2 It is a flowchart of a method for generating a live broadcast image based on a digital human provided according to another embodiment of the present disclosure.

[0052] like Figure 2 As shown, the method for generating a live broadcast image based on a digital human may include the following steps:

[0053] Step 201 : obtaining at least one first monitoring information selected from among crowd flow information of a target scene, operation data of an interactive device set in the target scene, a scene environment image, and environment quality information.

[0054] It should be noted that the execution process of step 201 can refer to the execution process of any embodiment of the present disclosure, and will not be described in detail here.

[0055] Step 202 : Based on the category to which the first monitoring information belongs, the first monitoring information is processed using a data processing method corresponding to the category to obtain processed second monitoring information.

[0056] In the disclosed embodiments, different categories of first monitoring information may have corresponding data processing methods, and the data processing method for a category may be determined based on the category to which the first monitoring information belongs. As an example, a correspondence between the category to which the information belongs and the data processing method may be pre-established and stored. After determining the category to which the first monitoring information belongs, the correspondence may be queried to determine the data processing method corresponding to the category.

[0057] In the embodiment of the present disclosure, after determining the data processing method corresponding to the category according to the category to which the first monitoring information belongs, the first monitoring information can be processed according to the data processing method corresponding to the category to obtain processed second monitoring information.

[0058] Optionally, in some embodiments, when the first monitoring information includes a scene environment image, image recognition may be performed on the scene environment image to obtain a scene description text, and the scene description text may be used as the second monitoring information.

[0059] As an example, for scene environment images, a frame capture strategy can be adopted to extract key image frames from the scene environment images, and / or a dynamic segmentation strategy can be adopted to convert the scene environment images into discrete video segments, and extract target video segments therefrom; then a multimodal visual large model is used to deeply analyze the target video segments and / or key image frames, and output an explanatory text, which is recorded as scene description text in this disclosure, and the explanatory text can be used as the second monitoring information.

[0060] As another example, for a scene environment image, a key image frame can be captured from the scene environment image, and an OCR (Optical Character Recognition) algorithm can be used to recognize the characters in the key image frame to obtain a scene description text, and the scene description text can be used as the second monitoring information. For example, in a scene where the target scene is a classroom, a video of the classroom teaching process can be obtained, and a key image frame with a blackboard as the object can be captured from the teaching process video. The OCR algorithm can be used to recognize the characters on the blackboard in the key image frame to obtain a scene description text, and the scene description text can be used as the second monitoring information.

[0061] Optionally, in some embodiments, when the first monitoring information includes crowd flow information and operation data, the crowd flow information and operation data can be converted into a first format to obtain scene infield data, and the scene infield data can be used as the second monitoring information.

[0062] As an example, the crowd flow information and the operation data may be processed into information in a first set format to obtain scene interior data, and the scene interior data may be used as the second monitoring information.

[0063] The first setting format is, for example, JSON (JavaScript Object Notation) format, XML (Extensible Markup Language) format, etc., which is not limited in the present disclosure.

[0064] For example, assuming that the first set format is JSON format, the crowd flow information and operation data are converted into JSON format strings, and the obtained scene in-field data is, for example: {"Total number of people in the scenic area": ​​"123", "User touch data": ["Question 1", "Question 2"], "Number of people in scenic spots": {"Scenic spot A": 10, "Scenic spot B": 20}}, and then, the scene in-field data can be used as the second monitoring information.

[0065] Optionally, in some embodiments, when the first monitoring information includes environmental quality information, the environmental quality information is converted into a second format to obtain target environmental information, and the target environmental information can be used as the second monitoring information.

[0066] As an example, the environmental quality information may be converted into information in a second set format to obtain target environmental information, and the target environmental information may be used as the second monitoring information.

[0067] The second setting format may be pre-set, such as a JSON format, etc., and the present disclosure does not impose any limitation on this.

[0068] For example, assuming that the second setting format is JSON format, the environmental quality information is converted into a string in JSON format, and the target environmental information obtained is, for example: {"temperature":"27","humidity":"40%","noise":"456","wind force":"level 7","light":"11.88"}, and then, the target environmental information can be used as the second monitoring information.

[0069] Therefore, according to the category to which the first monitoring information belongs, data processing of the first monitoring information can be achieved, thereby improving the accuracy and effectiveness of the processing of the first monitoring information.

[0070] It should be noted that the above-mentioned data processing methods for different categories of first monitoring data are only exemplary. In actual applications, other methods can be used. For example, when the first monitoring information includes a scene environment image, image recognition can be performed on the scene environment image to obtain a scene description text, and key text extraction can be performed on the scene description text to obtain a key description text, and the key description text can be used as the second monitoring information. The data processing methods corresponding to the above-mentioned different categories can be set according to actual conditions. For example, data preprocessing can also be performed on the environmental quality information, etc. The present disclosure does not impose any restrictions on this.

[0071] Step 203: Fill the second monitoring information into the filling position corresponding to the category in the prompt template to obtain the target prompt word.

[0072] It should be noted that the prompt template may contain fill-in positions corresponding to different categories. Therefore, in this embodiment, the second monitoring information can be filled into the fill-in position corresponding to the category of the first monitoring information corresponding to the second monitoring information in the prompt template to obtain the target prompt word.

[0073] Optionally, in some embodiments, target key information may be obtained; the second monitoring information and the target key information may be filled into corresponding filling positions in the prompt template respectively to obtain a target prompt word.

[0074] Among them, the target key information may include digital human character information, live broadcast interaction data, digital human language style information, content specification information of the explanation content, etc. It should be noted that the target key information can be added or set according to actual needs, and this disclosure does not impose any restrictions on this.

[0075] Among them, live broadcast interaction data can indicate the data generated by the interaction between the live broadcast audience and the live broadcast platform, such as data on questions, suggestions, requests, etc., which is not limited in this disclosure.

[0076] The digital human's language style information can be used to indicate the digital human's language style and to constrain the digital human's language style when explaining. In one example, the digital human's language style information is, for example:

[0077] The live broadcast style is friendly, infectious, interactive and positive;

[0078] Use natural spoken language, steady and solid tone, rigorous and serious;

[0079] In order to get closer to users, call them "this friend" or "this family member".

[0080] It should be noted that the above examples of the language style information of the digital human are merely exemplary. In actual applications, other examples may be used and can be set as needed.

[0081] The content specification information of the explanation content is the standardization requirements for the explanation content. For example, the content specification information of the explanation content is:

[0082] Note: Avoid markdown, line breaks, and list symbols in your replies.

[0083] Note: The entire output must not be segmented.

[0084] Among them, "avoid markdown, line breaks, and list symbols in replies" is used to constrain the format of the explanation content, and "output in whole paragraphs, without segmentation" is used to constrain the coherence of the explanation content.

[0085] It should be noted that the above examples of content specification information for explanation content are merely exemplary, and in actual applications, other information can be used and can be set as needed.

[0086] In an embodiment of the present disclosure, target key information can be obtained. In one example, the target key information can be directly read from local storage. In another example, the target key information can be obtained in response to input operations of relevant staff.

[0087] Regarding the acquisition of language style information of digital humans, in a possible implementation of the embodiment of the present disclosure, first tourists belonging to the target age group can be identified from the scene environment image; the number of first tourists can be obtained; and language style information can be obtained based on the number.

[0088] The target age groups may include children, teenagers, adults, and the elderly.

[0089] In one example, when the target age group includes children and the elderly, facial recognition and age analysis techniques can be used to identify children and the elderly from a specific image frame of the scene environment image. The number of children and the elderly is then counted to obtain the number of children and the number of elderly, respectively. Finally, language style information is determined based on the number of children and the number of elderly. For example, when the number of children is greater than the number of elderly, the language style is determined as cheerful and lively; when the number of elderly is greater than the number of children, the language style is determined as mature and steady, and so on. It should be noted that the above examples of language style information are merely exemplary and can be modified as needed in actual applications.

[0090] Therefore, with the help of automatic recognition technology, the number of tourists of each age group in the scene environment image can be automatically and accurately counted, and the language style of the digital human's explanation can be scientifically determined based on this, thereby improving the degree of match between the generated explanation content and the target audience; at the same time, appropriate language styles are selected according to the characteristics of different age groups to carry out explanations, thereby improving the match between them and user needs, which helps to improve the quality of live broadcast content.

[0091] In another possible implementation of the embodiment of the present disclosure, the emotional state of the second visitor in the scene environment image can be identified to obtain emotional state information; the second visitor is clustered based on the emotional state information to obtain at least one cluster; and the language style information is determined based on the number of visitors in the at least one cluster.

[0092] It should be noted that the present disclosure does not limit the number of second tourists.

[0093] The emotional state information may be used to indicate the emotional state of the corresponding visitor, wherein the emotional state may include but is not limited to excitement, pleasure, anxiety, calmness, etc.

[0094] In one example, a facial expression recognition algorithm and a body posture recognition algorithm can be combined to identify the emotional state of a tourist in a certain image frame of a scene environment image to obtain emotional state information. Second tourists can be clustered based on the emotional state information, that is, at least one cluster is generated based on second tourists with the same emotional state in the image frame, and the number of second tourists in the cluster can be counted. Finally, language style information is determined based on the number of tourists in each cluster and the emotional state corresponding to each cluster. For example, when the number of tourists in the cluster with the emotional state of "excited" or "happy" is the largest, the language style information can be positioned as energetic, cheerful, enthusiastic, and contagious; when the number of tourists in the cluster with the emotional state of "calm" or "relaxed" is the largest, the language style information can be positioned as soothing, gentle, and talkative, etc. It should be noted that the above examples of language style information are only exemplary and can be set as needed in actual applications.

[0095] Therefore, by making the live broadcast language style accurately fit the emotional state of tourists, the digital person's emotional guidance ability can be effectively enhanced, the match with user needs can be improved, and the user experience and the quality of live broadcast content can be improved.

[0096] Step 204: input the target prompt word into the language model to obtain the explanation content output by the language model.

[0097] Step 205: Generate a first live broadcast image of the digital human according to the explanation content.

[0098] Step 206: Fusing the scene environment image and the first live broadcast image of the digital human to obtain a second live broadcast image of the digital human live broadcast scene.

[0099] It should be noted that the execution process of steps 204 to 206 can refer to the execution process of any embodiment of the present disclosure and will not be described in detail here.

[0100] The method for generating live broadcast images based on a digital human in the disclosed embodiment processes the first monitoring information according to the category to which it belongs, using a data processing method corresponding to the category, to obtain processed second monitoring information; and then fills the second monitoring information into the corresponding fill position in the prompt template to obtain the target prompt word. This method can automatically process the data of the first monitoring information based on the category to which the monitoring information belongs, and accurately fill the corresponding position of the prompt template with the second monitoring information after data processing, thereby effectively organizing and integrating the information and constructing a clear and orderly information architecture. This eliminates the need for relevant staff to spend a lot of time and energy sorting out and integrating various information. Instead, they can quickly obtain the target prompt word by simply filling in the information according to the established template, thereby improving the efficiency of generating the target prompt word.

[0101] In order to clearly illustrate how the scene environment image and the first live broadcast image are fused to obtain the digital human scene environment image in any of the above embodiments of the present disclosure, the present disclosure also proposes a live broadcast image generation method based on the digital human.

[0102] Figure 3 It is a flowchart of a method for generating a live broadcast image based on a digital human provided according to another embodiment of the present disclosure.

[0103] like Figure 3 As shown, the method for generating a live broadcast image based on a digital human may include the following steps:

[0104] Step 301: Obtain at least one first monitoring information selected from the group consisting of crowd flow information of a target scene, operation data of an interactive device set in the target scene, a scene environment image, and environment quality information.

[0105] Step 302: Generate a target prompt word according to the first monitoring information.

[0106] Step 303: Input the target prompt word into the language model to obtain the explanation content output by the language model.

[0107] Step 304: Generate a first live broadcast image of the digital human according to the explanation content.

[0108] It should be noted that the execution process of steps 301 to 304 can refer to the execution process of any embodiment of the present disclosure and will not be described in detail here.

[0109] Step 305: Perform visual analysis on the scene environment image to obtain visual attractiveness information of the scene environment image.

[0110] Among them, the visual attractiveness information can be used to measure the attractiveness of the scene environment image, and the visual attractiveness information can be in the form of, but not limited to, numerical values, levels, etc. The form of the visual attractiveness information can be set according to actual needs, and this disclosure does not impose any restrictions on this. It should be noted that when the visual attractiveness information is in numerical form, its value space can be, for example, [0, 1]; when the visual attractiveness information is in the form of levels, its value space can be, for example, [low level, medium level, high level, highest level]. This disclosure does not impose any restrictions on the value space of the visual attractiveness information.

[0111] In a possible implementation of the embodiment of the present disclosure, a sub-index value of the scene environment image in a target dimension may be obtained; and visual attractiveness information may be determined based on the sub-index value of the target dimension.

[0112] Among them, the target dimensions may include live broadcast satisfaction dimension, picture quality dimension and content presentation characteristic dimension, etc., which can be added or set according to actual needs, and this disclosure does not impose any restrictions on this.

[0113] Optionally, in some embodiments, when the target dimension includes a live broadcast satisfaction dimension, the live broadcast comment data associated with the scene environment image in the live broadcast platform can be segmented to obtain multiple segmentations; the parts of speech of the multiple segmentations are obtained; and based on the parts of speech of the multiple segmentations, the sub-indicator values ​​of the live broadcast satisfaction dimension are determined.

[0114] The live broadcast comment data may be a collection of various text feedback information posted by viewers on the live broadcast platform regarding the live broadcast content during the live broadcast.

[0115] In an embodiment of the present disclosure, live comment data associated with a scene environment image can be obtained. In one example, live comment data of a live broadcast platform can be extracted in real time, and the publishing time of the live comment data can be compared with the timestamp of the scene environment image; when the publishing time of the live comment data is within the target time period corresponding to the timestamp of the scene environment image, it can be determined that the live comment data is associated with the scene environment image.

[0116] The target time period is determined based on the timestamp of the scene environment image and a set duration (eg, 0.5s, 1s, etc.) It should be noted that the present disclosure does not impose any restrictions on the value of the set duration.

[0117] Among them, the parts of speech of participles may include but are not limited to: positive adjectives (such as beautiful, satisfied, etc.), negative adjectives (such as bad, awful, etc.), positive action verbs (such as like, recommend, etc.), negative action verbs (such as complain, complain, etc.), positive degree adverbs used to modify positive adjectives or positive action verbs (such as very, extremely, etc.), negative degree adverbs used to modify negative adjectives or negative action verbs (such as very, too, etc.), and so on.

[0118] In one example, a word segmentation tool can be used to perform word segmentation processing on the live broadcast comment data associated with the scene environment image in the live broadcast platform to obtain multiple word segments, and the parts of speech of the multiple word segments can be identified. Then, based on the parts of speech of the multiple word segments, the sub-indicator value of the live broadcast satisfaction dimension can be determined. For example, the scores corresponding to various parts of speech can be pre-set. After determining the parts of speech of the multiple word segments, the scores corresponding to the parts of speech of the multiple word segments can be weighted and summed to obtain the target scores of the multiple word segments, and the target scores can be determined as the sub-indicator value of the live broadcast satisfaction dimension.

[0119] Therefore, by analyzing the parts of speech of the segmented words in the live broadcast comment data, we can more accurately capture the emotional tendencies of users watching the live broadcast towards the scene environment images, and then automatically and efficiently determine the sub-indicator values ​​of the live broadcast satisfaction dimension, thereby improving the accuracy of determining the sub-indicator values ​​of the live broadcast satisfaction dimension.

[0120] Optionally, in some embodiments, when the target dimension includes a picture quality dimension, a sub-index value of the picture quality dimension may be determined based on an evaluation index of the scene environment image.

[0121] Among them, the evaluation indicators may include the matching degree between multiple colors contained in the scene environment image, the matching degree between the saturation of the scene environment image and the scene type to which the target scene belongs, the stability of the scene environment image, the matching degree between the saturation of the scene environment image and the scene type to which the target sub-scene associated with the scene environment image belongs, etc., which can be added according to actual needs and are not limited in this disclosure.

[0122] The matching degree between the multiple colors contained in the scene environment image can be used to measure the matching degree between the multiple colors contained in the scene environment image, or can be used to indicate whether the multiple colors contained in the scene environment image match each other.

[0123] Among them, the scene type can be but not limited to natural scenery, temples, museums, culture and art, etc., and can be set and added according to actual needs. This disclosure does not impose any restrictions on this.

[0124] The matching degree between the saturation of the scene environment image and the scene type to which the target scene belongs may be used to indicate whether the saturation of the scene environment image matches the scene type to which the target scene belongs.

[0125] Among them, the stability of the scene environment image can be used to measure the stability of the scene environment image.

[0126] The target sub-scene may be a part of the target scene, for example, a scenic spot in a scenic area is a part of the target scene.

[0127] The degree of matching between the saturation of the scene environment image and the scene type to which the target subscene associated with the scene environment image belongs can be used to measure whether the saturation of the scene environment image and the scene type to which the target subscene associated with the scene environment image belongs match.

[0128] Optionally, in some embodiments, the key image frame in the scene environment image can be converted into the HSV (Hue, Saturation, Value) space to obtain a target image; the colors are classified according to the hue value of each pixel in the target image to obtain multiple categories; the number of pixels belonging to any category is counted to determine the main color, auxiliary color and accent color based on the number of pixels in each category; then, based on color theory, it is determined whether the main color, auxiliary color and accent color are reasonably matched; when it is determined that the main color, auxiliary color and accent color are reasonably matched, the matching between the multiple colors contained in the scene environment image is determined.

[0129] Optionally, in some embodiments, a correspondence between scene type and saturation can be pre-constructed and saved. Then, after determining the saturation of the scene environment image, the correspondence can be queried to determine whether the saturation of the scene environment image matches the scene type to which the target scene belongs.

[0130] Optionally, in some embodiments, a video stabilization detection tool may be used to analyze the jitter amplitude and jitter frequency of the scene environment image, and then determine the stability of the scene environment image based on the jitter amplitude and jitter frequency.

[0131] In one example, the index values ​​of the evaluation indexes of the scene environment image may be weighted and summed to obtain a sub-index value of the picture quality level.

[0132] Therefore, by accurately capturing the color coordination of the scene environment image, the adaptability of the color and scene type, and the smoothness of the picture playback, a basis is provided for determining the picture quality level of the scene environment image, and then, the quantitative evaluation and precise control of the picture quality dimensions of the scene environment image are effectively realized.

[0133] Optionally, in some embodiments, when the target dimension includes a content presentation characteristic dimension, a video understanding model can be used to obtain a video description text of the scene environment image; the matching degree between the video description text and the characteristic elements of the target scene is analyzed to obtain a matching score; and based on the matching score, the sub-indicator value of the content presentation characteristic dimension is determined.

[0134] Among them, the video understanding model can have the function of understanding the video content.

[0135] Among them, characteristic elements may include but are not limited to landform features (such as mountains, caves, etc.), water landscapes (such as waterfalls, etc.), biological landscapes (such as characteristic animals, rare plants, etc.), historical relics (such as ancient buildings, ancient city walls, etc.), national culture (such as traditional festivals, special food, etc.), special activities, interactive experience projects, characteristic buildings, etc., which can be set or added according to actual needs, and this disclosure does not impose any restrictions on this.

[0136] In one example, after using a video understanding model to obtain video description text of a scene environment image, natural language analysis technology can be used to determine the semantic similarity between the video description text and the text corresponding to the characteristic elements of the target scene, so as to evaluate the matching degree between the video description text and the characteristic elements of the target scene based on the semantic similarity; and determine the semantic similarity as the matching score between the video description text and the characteristic elements of the target scene; and then, determine the matching score as a sub-indicator value of the content presentation characteristic dimension.

[0137] Therefore, with the help of the powerful learning ability of the video understanding model, it is possible to comprehensively and accurately capture the key information in the scene environment image, and then generate rich and detailed video description text; matching analysis of the video description text and the characteristic elements of the target scene can deeply explore the degree of fit between the scene live broadcast content and the characteristics of the target scene, and provide a basis for determining the content presentation characteristic dimensions of the scene environment image, thereby realizing quantitative evaluation and precise control of the content presentation characteristic dimensions of the scene environment image.

[0138] In the disclosed embodiments, after obtaining the sub-indicator values ​​for each target dimension of the scene environment image, visual attractiveness information can be determined based on the sub-indicator values ​​for each target dimension. For example, a weighted sum of the sub-indicator values ​​for each target dimension can be performed to obtain a target score, which can then be used as the visual attractiveness information. Alternatively, a corresponding attractiveness level can be determined based on the target score, and the attractiveness level can be used as the visual attractiveness information.

[0139] Therefore, the visual attractiveness information of the scene environment image can be accurately and effectively determined with the help of the sub-index value of the target dimension.

[0140] Step 306: Determine a fusion strategy for the scene environment image and the first live broadcast image based on the visual attractiveness information.

[0141] Among them, the screen fusion strategy may include screen display ratio, screen position coordinates, etc., which can be set or added according to actual needs, and the present disclosure does not impose any restrictions on this.

[0142] In one example, a corresponding mapping relationship between visual attractiveness information and image fusion strategy can be pre-set. In subsequent applications, after determining the visual attractiveness information, the image fusion strategy of the scene environment image and the first live broadcast image of the digital human can be determined based on the visual attractiveness information and the mapping relationship.

[0143] Step 307: The scene environment image and the first live broadcast image of the digital human are fused according to the image fusion strategy to obtain a second live broadcast image of the digital human live broadcast scene.

[0144] Optionally, in some embodiments, when the screen display strategy includes a first screen display ratio, the sizes of the first live broadcast image and the scene environment image can be adjusted based on the first screen display ratio; the adjusted first live broadcast image and the scene environment image are fused to obtain a second live broadcast image. In one example, when the visual appeal information is a visual appeal score, when the visual appeal score is high, indicating a high level of appeal for the scene environment image, the size of the scene environment image can be increased and the size of the digital human's first live broadcast image can be reduced based on the first screen display ratio; when the visual appeal score is low, indicating a low level of appeal for the scene environment image, the size of the digital human's first live broadcast image can be increased and the size of the scene environment image can be reduced based on the first screen display ratio.

[0145] Therefore, the display ratio of the scene environment image and the first live broadcast image of the digital human can be adaptively adjusted based on the visual attractiveness information, and then the size of the scene environment image and the first live broadcast image can be adjusted. This can concentrate limited live broadcast resources on the corresponding images, avoid resource waste caused by fixed screen layout, and improve the utilization efficiency of live broadcast resources.

[0146] It is understandable that during the live broadcast process, if the second live broadcast image is always presented at a fixed screen display ratio, as the live broadcast time goes by, users watching the live broadcast are very likely to gradually develop visual fatigue.

[0147] Optionally, in some embodiments, the adjustment range of the first screen display ratio can be determined based on the first screen display ratio; within the adjustment range, the first screen display ratio is adjusted according to the live broadcast duration to obtain the second screen display ratio; based on the second screen display ratio, the scene environment image and the first live broadcast image are reintegrated to update the second live broadcast image.

[0148] It should be noted that the first screen display ratio can have a corresponding adjustment range. For example, when the first screen display ratio is 1:3, the corresponding adjustment range is 2:7 to 3:7. It should be noted that the above examples of the adjustment range corresponding to the first screen display ratio are only exemplary. In actual applications, the adjustment range corresponding to the first screen display ratio can be set according to actual needs, and this disclosure does not impose any restrictions on this.

[0149] In one example, a correspondence between the screen display ratio and the adjustment range may be pre-established and saved, and then when the first screen display ratio is determined, the adjustment range corresponding to the first screen display ratio may be determined.

[0150] In an embodiment of the present disclosure, the display ratio of the first screen can be adjusted according to the live broadcast duration within the adjustment range to obtain the display ratio of the second screen. In one example, when the live broadcast duration gradually meets at least one set duration, the live broadcast duration can be gradually adjusted from the minimum value in the adjustment range to the maximum value in the adjustment range, or the live broadcast duration can be gradually adjusted from the maximum value in the adjustment range to the minimum value in the adjustment range. It should be noted that the set duration can be pre-set, and the present disclosure does not limit its value. For example, assuming the adjustment range is 2:7 to 3:7, and the set durations include 5 minutes, 10 minutes, and 16 minutes, when the live broadcast duration gradually meets 5 minutes, 10 minutes, and 16 minutes, the display ratio of the first screen can be adjusted. For example, the display ratio of the second screen gradually changes to 2:7, 1:3, and 3:7. That is, when the live broadcast duration meets 5 minutes, the display ratio of the first screen is adjusted to 2:7, when the live broadcast duration meets 10 minutes, the display ratio of the second screen is adjusted to 1:3, and when the live broadcast duration meets 16 minutes, the display ratio of the first screen is adjusted to 3:7.

[0151] In this way, the screen display ratio of the scene environment image and the first live broadcast image of the digital human can be flexibly adjusted, avoiding the visual fatigue problem caused by a fixed screen display ratio. At the same time, the adjustment range of the screen display ratio is determined in advance, providing a clear operating boundary for the screen ratio adjustment, avoiding excessive adjustment that affects the overall effect of the live broadcast and the audience's viewing experience.

[0152] The method for generating live images based on digital humans in the disclosed embodiments visually analyzes a scene environment image to obtain visual attractiveness information for the scene environment image; based on this visual attractiveness information, a fusion strategy is determined for the scene environment image and a first live image; and the scene environment image and the first live image are fused according to the fusion strategy to obtain a second live image. Thus, formulating a fusion strategy based on the visual attractiveness information of the scene environment image can better integrate the real scene with the virtual digital human, improving the flexibility and effectiveness of the image fusion process.

[0153] In order to clearly illustrate how the first live broadcast image of the digital human is generated according to the explanation content in the above embodiment of the present disclosure, the present disclosure also proposes a live broadcast image generation method based on the digital human.

[0154] Figure 4 It is a flowchart of a method for generating a live broadcast image based on a digital human provided according to another embodiment of the present disclosure.

[0155] like Figure 4 As shown, the method for generating a live broadcast image based on a digital human may include the following steps:

[0156] Step 401: Obtain at least one first monitoring information selected from the group consisting of crowd flow information of a target scene, operation data of an interactive device set in the target scene, a scene environment image, and environment quality information.

[0157] Step 402: Generate a target prompt word according to the first monitoring information.

[0158] Step 403: Input the target prompt word into the language model to obtain the explanation content output by the language model.

[0159] It should be noted that the execution process of step 401 to step 402 can refer to the execution process of any embodiment of the present disclosure, and will not be described in detail here.

[0160] Step 404: convert the explanation content into voice information.

[0161] As an example, a TTS (Text-to-Speech) tool may be used to convert the explanation content into a voice signal, which is recorded as voice information in this disclosure.

[0162] Step 405: Input the voice information into the digital human driving model to obtain digital population data.

[0163] Among them, digital population data can be used to describe and control a series of data information about the lip movement state of the digital person during the speaking process.

[0164] It should be noted that the digital human driving model can generate digital population data based on the input voice information.

[0165] Optionally, the digital human driving model can also generate multimodal data such as the digital human's movements, facial expressions, body postures, etc. based on the input voice information.

[0166] Step 406: Render the digital human according to the digital population data to obtain a first live broadcast image.

[0167] Optionally, in some embodiments, the corresponding first digital human expression data can be obtained based on the environmental quality information, and then the digital human can be rendered based on the digital population data and the first digital human expression data to obtain the first live broadcast image.

[0168] The first digital human expression data may include digital human facial expression information and digital human movement information, etc., which can be added or set according to actual needs, and the present disclosure does not impose any restrictions on this.

[0169] In the embodiment of the present disclosure, the corresponding first digital human expression data can be obtained according to the environmental quality information.

[0170] In one example, when the environmental quality information includes meteorological information, meteorological information for the current moment and the moment before the current moment can be obtained, and a comparative analysis can be performed on the current and previous meteorological information. Based on the comparative analysis results, the first digital human's expression data can be determined. For example, if the current meteorological information indicates cloudy, and the previous meteorological information indicates raining, a comparative analysis can be performed to determine that the weather has deteriorated. Furthermore, based on the comparative analysis results, the digital human's expression information can be determined to be "nervous" and the digital human's movement information can be determined to be "running."

[0171] In another example, if the environmental quality information includes light intensity, the light intensity at the current moment and the light intensity at the moment before the current moment can be obtained, and a comparative analysis can be performed on the light intensity at the current moment and the light intensity at the moment before the current moment to determine the first digital human's demeanor data based on the comparative analysis results. For another example, if the light intensity at the current moment is 0.5 and the light intensity at the moment before the current moment is 1000, it indicates a sudden drop in light intensity (such as the sun setting). Then, the first digital human's demeanor data can be determined based on the comparative analysis results.

[0172] Therefore, rendering the digital human based on the expression data obtained from environmental quality information can enable the digital human to break away from the mechanical stereotyped image, improve the adaptability between the digital human's expression and the target scene, and thus improve the user experience and the quality of the live broadcast content.

[0173] Optionally, in some embodiments, action labels can be obtained based on the explanation content through a language model, and then the digital human can be rendered based on the digital human population data and the digital human action data corresponding to the action labels to obtain the first live broadcast image.

[0174] Among them, the action label can be used to indicate the corresponding action.

[0175] Among them, the language model can be a large model, which has the function of identifying actions that match the text and outputting corresponding labels (i.e., action labels) based on the identified actions. In one example, the target information corresponding to the text can be identified by the language model to determine the corresponding action label based on the target information. Among them, the target information can be, for example, but not limited to, intention, emotion (or mood) information, keywords related to the action, etc., and the present disclosure does not impose any restrictions on this. For example, the text is "This mobile phone has a battery life of up to 48 hours", and its intention is to promote the advantages of the product, and the corresponding action label can be "thumbs up".

[0176] The action tag may have corresponding digital human action data. In one example, a correspondence between the action tag and the digital human action data may be pre-established, and then, when the action tag is determined, the correspondence may be queried to obtain the corresponding digital human action data.

[0177] Therefore, with the help of the powerful computing power of the language model, it is possible to accurately extract action information based on the explanation content, obtain action labels, and provide clear guidance for the digital human's action performance; combining the digital population data and the digital human action data corresponding to the action labels for rendering processing, improve the matching degree between the digital human's actions and the explanation content, thereby improving the presentation effect of the digital human live broadcast and the quality level of the live broadcast content.

[0178] Step 407: Fusing the scene environment image and the first live broadcast image of the digital human to obtain a second live broadcast image of the digital human live broadcast scene.

[0179] It should be noted that the execution process of step 407 can refer to the execution process of any embodiment of the present disclosure, and will not be described in detail here.

[0180] Optionally, in some embodiments, when the environmental quality information includes meteorological information, a corresponding digital human background image may be determined based on the meteorological information, and the digital human background image may be integrated into the first live broadcast image.

[0181] In one example, key meteorological information may be extracted from the meteorological information, and based on the key meteorological information, a corresponding digital human background image may be determined for integration into the first live broadcast image.

[0182] Among them, key meteorological information may include weather types (such as rainy days, snowy days, windy days, sunny days, cloudy days, etc.) and corresponding levels (such as light rain, heavy rain, rainstorms, thunderstorms, etc. in rainy days), etc., which can be added or set according to actual needs, and this disclosure does not impose any restrictions on this.

[0183] Therefore, by determining the corresponding digital human background image based on meteorological information, the digital human live broadcast scene can be closely aligned with the real environment, enhancing the realism of the digital human live broadcast, thereby improving the user experience and the quality of the live broadcast content.

[0184] Optionally, in some embodiments, illumination parameters and time information are obtained based on the scene environment image; and according to the illumination parameters and time information, the picture rendering parameters of the first live image are adjusted.

[0185] The lighting parameters may include but are not limited to: lighting intensity, lighting direction, etc.

[0186] Among them, the picture rendering parameters can be used to control various adjustable variables of the image or video output effect, such as including but not limited to: image brightness, color saturation, etc., which can be added or set according to actual needs, and this disclosure does not impose any restrictions on this.

[0187] In one example, a visual algorithm can be used to perform image analysis on a scene environment image to obtain lighting parameters, and time information can be obtained based on the timestamp of the scene environment image; based on a pre-established mapping relationship between the lighting parameters and time information and the picture rendering parameters, the corresponding picture rendering parameters are determined, and then the first live image is adjusted based on the picture rendering parameters.

[0188] Therefore, the rendering parameters of the first live broadcast image can be dynamically adjusted based on the lighting parameters and time information, thereby enhancing the adaptability of the digital human's presentation effect to specific time periods and lighting conditions, improving the realism and naturalness of the digital human's live broadcast, and thus improving the quality of the live broadcast content.

[0189] Optionally, in some embodiments, a live video may be generated based on the voice information and the second live image. In one example, the voice information and the second live image may be aligned to obtain the live video.

[0190] Optionally, in some embodiments, when the environmental quality information includes the ambient noise decibels of the scene, the volume of the digital human's voice can be controlled based on the ambient noise decibels. For example, when the ambient noise decibels are high, the volume of the digital human's voice can be controlled to be louder.

[0191] Therefore, dynamic control of the digital human's voice volume based on the decibels of ambient noise can make the sound emitted by the digital human match the surrounding environment, greatly optimizing the auditory experience of the digital human's live broadcast.

[0192] The method for generating live broadcast images based on digital humans in the disclosed embodiments converts explanation content into voice information; inputs the voice information into a digital human driving model to obtain digital population data; and renders the digital human based on the digital population data to produce a first live broadcast image. Thus, the explanation content is converted into voice information for transmission, which is then input into the digital human driving model to obtain lip data, which is then rendered accordingly, achieving synchronization between the image and the voice. This method automatically and efficiently converts the explanation content into the first live broadcast image of the digital human, greatly improving the efficiency of live broadcast content production, reducing labor costs, and enhancing the consistency and professionalism of the digital human live broadcast.

[0193] The following describes the method for generating live images based on digital humans in a scenic area as an example. Figure 5 FIG. 5 is a schematic diagram of the structure of a live broadcast image generation system based on a digital human. The live broadcast image generation system 50 based on a digital human may include a data acquisition layer 51, a data processing layer 52, a content generation layer 53, a real-time rendering layer 54, and a user interaction layer 55; wherein:

[0194] Data collection layer 51:

[0195] The data of the scenic area can be collected by multiple monitoring devices deployed in the scenic area. The multiple monitoring devices and the data collected can be:

[0196] 1) 360° panoramic camera: can be deployed at key points in scenic areas (such as entrances and viewing platforms), supports 8K resolution, HDR dynamic exposure, and real-time streaming via the RTMP protocol; used to capture scenic area video streams (referred to as scene environment images in this disclosure);

[0197] 2) Environmental sensors:

[0198] Crowd density detection equipment: millimeter-wave radar and cameras work together to achieve intelligent monitoring of crowd density or population size in scenic spots or attractions;

[0199] Weather station: used to collect environmental information such as temperature, air humidity, rainfall information, wind speed, etc. in the scenic area (referred to as environmental quality information in this disclosure);

[0200] Acoustic sensor: used to identify the decibels of ambient noise;

[0201] 3) IoT devices: scenic area gates (used to record the number of people entering the park in real time), smart guide screens (used to obtain visitor touchpoint data (recorded as operation data in this disclosure) (such as visitor click query information, etc.)).

[0202] Data processing layer 52:

[0203] Processing the collected data

[0204] 1) For the real-time scenic spot video stream collected, split it into small video segments or intercept image frames, use the multimodal visual model to understand the image / video, and output the real-time scenic spot video stream (referred to as scene description text in this disclosure)

[0205] 2) Process the environmental information into a unified format and convert it into a JSON string. The environmental information may include scenic area temperature, air humidity, rainfall information, wind speed, ambient noise decibels, etc. The string may be, for example, {"Scenic area temperature": 27, "Scenic area humidity": "xx", "Noise": "xx", "Wind intensity": "xx", "Light intensity": "xx"};

[0206] 3) Convert the crowd flow information and visitor touchpoint data into a unified data format. The crowd flow information may include the number of park visitors and the number of visitors to scenic spots, and the visitor touchpoint data may include visitor click query information. For example, the crowd flow information and visitor touchpoint data after format conversion are: {"Total number of visitors to scenic spots": "123", "User query": ["Question 1", "Question 2"], "Number of visitors to scenic spots": {"Scenic spot A": 10, "Scenic spot B": 20}};

[0207] 4) Collect questions from viewers in the live broadcast room (referred to as live broadcast interaction data in this disclosure)

[0208] Content Generation Layer 53:

[0209] The data processed by the data processing layer 52 is then filled into the corresponding fill-in position in the prompt template to obtain the target prompt word. It should be noted that the prompt template may also contain fill-in positions for the digital human's character information (referred to as digital human role information in this disclosure), the digital human's language style information, and notes (referred to as content specification information for the explanation content in this disclosure).

[0210] For example, the prompt template is:

[0211]

[0212]

[0213] Optionally, tourists belonging to a target age group are identified from a video stream of a scenic spot, and the number of identified tourists belonging to the target age group is counted to obtain the number of tourists belonging to the target age group; and language style information is obtained based on the number of tourists belonging to the target age group.

[0214] Real-time rendering layer 54:

[0215] 1) For the explanation content returned by the large model, the explanation content is first converted into a voice signal (referred to as voice information in this disclosure) through the TTS algorithm, and then converted into an image through the digital human driving model, and the image is encoded into a digital human video stream (referred to as the first live image in this disclosure).

[0216] Optionally, illumination information (referred to as illumination parameters in this disclosure) and time information may be obtained based on the scenic area video stream; and the brightness of the digital human video stream may be adjusted based on the illumination information and time information.

[0217] Optionally, the large model can output action tags based on the explanation content, and the corresponding action baseplate (referred to as digital human action data in this disclosure) can be determined based on the action tags. Then, the digital human video stream can be rendered based on the action baseplate corresponding to the action tags.

[0218] For example, the large model returns the action label and explanation content based on the explanation content as follows:

[0219]

Welcome Action

[0220] Based on the [Welcome Action] tag, the corresponding welcome action baseplate is matched to render the digital human video stream based on the action baseplate.

[0221] 2) Unify and fuse the digital human video stream and the scenic area video stream. Specifically, based on the visual appeal score of the scenic area video stream, a fusion strategy for the digital human video stream and the scenic area video stream is determined (referred to as the image fusion strategy in this disclosure). Then, based on this fusion strategy, OpenCV image processing technology is used to fuse the digital human video stream and the scenic area video stream, ultimately outputting a live video stream of the digital human scene (referred to as the second live image in this disclosure) to the user.

[0222] a. When the fusion strategy includes the screen display ratio, the display sizes of the digital human video stream and the scenic area video stream are adjusted according to the screen display ratio; the adjusted digital human video stream and the scenic area video stream are fused to obtain the digital human scene live video stream.

[0223] b. Random rotation adjustment strategy. The screen ratio will be randomly changed over time to avoid audience fatigue.

[0224] Optionally, the adjustment range of the screen display ratio is determined according to the screen display ratio; within the adjustment range, the screen display ratio is adjusted according to the live broadcast duration, and the digital human video stream and the scenic area video stream are re-integrated based on the adjusted screen display ratio to update the digital human scene live broadcast video stream.

[0225] User Interaction Layer 55:

[0226] Viewers can interact normally in the live broadcast room with barrages, such as asking questions, making comments, etc.

[0227] In summary, the disclosed method for generating live broadcast images based on digital humans, through the use of language models to generate explanations based on multimodal data, can dynamically adapt to the needs of different audience user groups, enabling in-depth explanations of scenes using the digital human. By integrating the scene environment image with the live broadcast image of the digital human, this method organically combines intuitive presentation with in-depth explanations, avoiding the drawbacks of traditional live broadcasts, such as the lack of content in purely visual presentations or the detached explanations from the actual scene. This method effectively improves the quality of live broadcast images, enhancing the richness and real-time nature of the image content.

[0228] In order to implement the above embodiment, the present disclosure also provides a live broadcast image generation device based on digital human. Figure 6 As shown, Figure 6 6 is a schematic diagram of a digital human-based live broadcast image generation device according to another embodiment of the present disclosure. The digital human-based live broadcast image generation device 60 may include: an acquisition module 601, a prompt module 602, a processing module 603, a generation module 604, and a fusion module 605.

[0229] The acquisition module 601 is configured to acquire at least one first monitoring information selected from among crowd flow information of a target scene, operation data of an interactive device provided in the target scene, a scene environment image, and environment quality information;

[0230] The prompt module 602 is configured to generate a target prompt word according to the first monitoring information.

[0231] The processing module 603 is used to input the target prompt word into the language model to obtain the explanation content output by the language model.

[0232] The generating module 604 is used to generate a first live broadcast image of the digital human according to the explanation content.

[0233] The fusion module 605 is used to fuse the scene environment image and the first live broadcast image of the digital human to obtain a second live broadcast image of the digital human live broadcast scene.

[0234] As a possible implementation method of the embodiment of the present disclosure, the prompt module 602 is used to: process the first monitoring information according to the category to which the first monitoring information belongs, using a data processing method corresponding to the category to obtain processed second monitoring information; fill the second monitoring information into the filling position corresponding to the category in the prompt template to obtain the target prompt word.

[0235] As a possible implementation method of an embodiment of the present disclosure, the prompt module 602 is used to: when the first monitoring information includes a scene environment image, perform image recognition on the scene environment image to obtain a scene description text, and use the scene description text as the second monitoring information; when the first monitoring information includes crowd flow information and operation data, perform a first format conversion on the crowd flow information and the operation data to obtain scene interior data, and use the scene interior data as the second monitoring information; when the first monitoring information includes environmental quality information, perform a second format conversion on the environmental quality information to obtain target environment information, and use the target environment information as the second monitoring information.

[0236] As a possible implementation of an embodiment of the present disclosure, the prompt module 602 is used to: obtain target key information; fill the second monitoring information and the target key information into the corresponding filling positions in the prompt template to obtain the target prompt word; wherein the target key information includes at least one of the following: digital human character information; live broadcast interaction data; digital human language style information; content specification information of the explanation content.

[0237] As a possible implementation method of the embodiment of the present disclosure, the target key information includes language style information, and the prompt module 602 is used to: identify the first tourist belonging to the target age group from the scene environment image; obtain the number of the first tourists; and obtain language style information based on the number.

[0238] As a possible implementation method of an embodiment of the present disclosure, the target key information includes language style information, and the prompt module 602 is used to: identify the emotional state of the second tourist in the scene environment image to obtain emotional state information; cluster the second tourist based on the emotional state information to obtain at least one cluster; and determine the language style information based on the number of tourists in the at least one cluster.

[0239] As a possible implementation method of the embodiment of the present disclosure, the fusion module 605 is used to: perform visual analysis on the scene environment image to obtain visual attractiveness information of the scene environment image; determine the image fusion strategy of the scene environment image and the first live image based on the visual attractiveness information; and fuse the scene environment image and the first live image according to the image fusion strategy to obtain a second live image.

[0240] As a possible implementation method of an embodiment of the present disclosure, the fusion module 605 is used to: obtain sub-indicator values ​​of the scene environment image in the target dimension; wherein the target dimension includes at least one of the live broadcast satisfaction dimension, the picture quality dimension and the content presentation characteristic dimension; and determine the visual attractiveness information based on the sub-indicator values ​​of the target dimension.

[0241] As a possible implementation method of the embodiment of the present disclosure, the target dimension includes a live broadcast satisfaction dimension, and the fusion module 605 is used to: perform word segmentation processing on the live broadcast comment data associated with the scene environment image in the live broadcast platform to obtain multiple word segments; obtain the parts of speech of the multiple word segments; and determine the sub-indicator value based on the part of speech.

[0242] As a possible implementation method of an embodiment of the present disclosure, the target dimension includes a picture quality dimension, and the fusion module 605 is used to: determine the sub-indicator value of the picture quality dimension based on the evaluation index of the scene environment image; the evaluation index includes at least one of the following: the matching degree between multiple colors contained in the scene environment image; the matching degree between the saturation of the scene environment image and the scene type to which the target scene belongs; the stability of the scene environment image.

[0243] As a possible implementation method of the embodiment of the present disclosure, the target dimension includes the content presentation characteristics dimension, and the fusion module 605 is used to: use the video understanding model to obtain the video description text of the scene environment image; analyze the matching degree between the video description text and the characteristic elements of the target scene to obtain a matching score; and determine the sub-indicator value of the content presentation characteristics dimension based on the matching score.

[0244] As a possible implementation method of the embodiment of the present disclosure, the screen fusion strategy includes a first screen display ratio; a fusion module 605 is used to: adjust the size of the first live image and the scene environment image according to the first screen display ratio; and fuse the adjusted first live image and the scene environment image to obtain a second live image.

[0245] As a possible implementation method of the embodiment of the present disclosure, the fusion module 605 is further used to: determine an adjustment range of the first screen display ratio based on the first screen display ratio; within the adjustment range, adjust the first screen display ratio according to the live broadcast duration to obtain a second screen display ratio; and re-fuse the scene environment image and the first live broadcast image based on the second screen display ratio to update the second live broadcast image.

[0246] As a possible implementation of the embodiment of the present disclosure, the generation module 604 is used to: convert the explanation content into voice information; input the voice information into the digital human driving model to obtain digital population data; and render the digital human according to the digital population data to obtain a first live broadcast image.

[0247] As a possible implementation of the embodiment of the present disclosure, the generation module 604 is configured to: obtain corresponding first digital human demeanor data based on environmental quality information; wherein the first digital human demeanor data includes at least one of digital human expression information and digital human movement information; and render the digital human based on the digital human population data and the first digital human demeanor data to obtain a first live broadcast image.

[0248] As a possible implementation of the embodiment of the present disclosure, the generation module 604 is used to: obtain action labels based on the explanation content through a language model; and render the digital human based on the digital population data and the digital human action data corresponding to the action labels to obtain a first live broadcast image.

[0249] As a possible implementation of the embodiment of the present disclosure, the environmental quality information includes meteorological information; the digital human-based live image generation device 60 may further include:

[0250] The first determining module is used to determine the corresponding digital human background image according to meteorological information.

[0251] The integration module is used to integrate the digital human background image into the first live broadcast image.

[0252] As a possible implementation of the embodiment of the present disclosure, the digital human-based live broadcast image generation device 60 may further include:

[0253] The second determination module is used to obtain lighting parameters and time information based on the scene environment image.

[0254] The adjustment module is used to adjust the picture rendering parameters of the first live image according to the lighting parameters and time information.

[0255] The live broadcast image generation device based on a digital human in the disclosed embodiment obtains at least one first monitoring information selected from the following: crowd flow information of a target scene, operation data of an interactive device set in the target scene, scene environment images, and environmental quality information; generates a target prompt word based on the first monitoring information; inputs the target prompt word into a language model to obtain explanation content output by the language model; generates a first live broadcast image of the digital human based on the explanation content; and fuses the scene environment image and the first live broadcast image of the digital human to obtain a second live broadcast image of the digital human live broadcast scene. Thus, the explanation content generated by the language model based on multimodal data can dynamically adapt to the needs of different user groups, thereby providing in-depth explanations of the scene based on the digital human; thereby, the scene environment image and the live broadcast image of the digital human are integrated to achieve an organic combination of intuitive display and in-depth explanation, avoiding the drawbacks of traditional live broadcasts where the simple visual display lacks connotation or the simple explanation is divorced from the actual scene, and can effectively improve the quality of the live broadcast image and enhance the richness and real-time nature of the image content.

[0256] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are all carried out with the user's consent, comply with relevant laws and regulations, and do not violate public order and good morals.

[0257] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0258] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0259] like Figure 7As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0260] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0261] The computing unit 701 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the method for generating a live broadcast image based on a digital human. For example, in some embodiments, the method for generating a live broadcast image based on a digital human can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the method for generating a live broadcast image based on a digital human described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the digital human-based live image generation method in any other appropriate manner (for example, by means of firmware).

[0262] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0263] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0264] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0265] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0266] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0267] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0268] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0269] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for generating live broadcast images based on digital humans, wherein: The method comprises: Acquire at least one first monitoring information selected from among crowd flow information of a target scene, operation data of an interactive device set in the target scene, a scene environment image, and environment quality information; generating a target prompt word according to the first monitoring information; Inputting the target prompt word into a language model to obtain explanation content output by the language model; generating a first live broadcast image of the digital human according to the explanation content; The scene environment image and the first live broadcast image of the digital human are fused to obtain a second live broadcast image of the digital human live broadcast scene.

2. The method according to claim 1, wherein Generating a target prompt word according to the first monitoring information includes: processing the first monitoring information according to the category to which the first monitoring information belongs, using a data processing method corresponding to the category, to obtain processed second monitoring information; The second monitoring information is filled into the filling position corresponding to the category in the prompt template to obtain the target prompt word.

3. The method according to claim 2, wherein: The step of processing the first monitoring information according to the category to which the first monitoring information belongs, using a data processing method corresponding to the category to obtain processed second monitoring information, includes: In a case where the first monitoring information includes the scene environment image, performing image recognition on the scene environment image to obtain scene description text, and using the scene description text as the second monitoring information; In a case where the first monitoring information includes the crowd flow information and the operation data, performing a first format conversion on the crowd flow information and / or the operation data to obtain scene infield data, and using the scene infield data as the second monitoring information; In a case where the first monitoring information includes the environmental quality information, the environmental quality information is converted into a second format to obtain target environmental information, and the target environmental information is used as the second monitoring information.

4. The method according to claim 2, wherein: Filling the second monitoring information into the filling position corresponding to the category in the prompt template to obtain the target prompt word includes: Obtain key target information; Filling the second monitoring information and the target key information into corresponding filling positions in the prompt template to obtain the target prompt word; The target key information includes at least one of the following: Digital human character information; Live interactive data; Language style information of digital humans; Content specification information for the explanatory content.

5. The method according to claim 4, wherein The target key information includes the language style information, and obtaining the target key information includes: identifying a first tourist belonging to a target age group from the scene environment image; Get the number of the first visitors; The language style information is acquired according to the quantity.

6. The method according to claim 4, wherein: The target key information includes the language style information, and obtaining the target key information includes: Identifying the emotional state of the second visitor in the scene environment image to obtain emotional state information; Clustering the second tourists based on the emotional state information to obtain at least one cluster; The language style information is determined based on the number of visitors in the at least one cluster.

7. The method according to claim 1, wherein The fusing of the scene environment image and the first live broadcast image of the digital human to obtain a second live broadcast image of the digital human live broadcast scene includes: Performing visual analysis on the scene environment image to obtain visual attractiveness information of the scene environment image; Determining, based on the visual attractiveness information, a screen fusion strategy for the scene environment image and the first live broadcast image; The scene environment image and the first live image are fused according to the image fusion strategy to obtain the second live image.

8. The method according to claim 7, wherein The performing visual analysis on the scene environment image to obtain visual attractiveness information of the scene environment image includes: Obtaining a sub-index value of the scene environment image in a target dimension; wherein the target dimension includes at least one of a live broadcast satisfaction dimension, a picture quality dimension, and a content presentation characteristic dimension; The visual attractiveness information is determined according to the sub-indicator value of the target dimension.

9. The method according to claim 8, wherein The target dimension includes the live broadcast satisfaction dimension, and obtaining the sub-index value of the scene environment image in the target dimension includes: Performing word segmentation processing on live comment data associated with the scene environment image in the live platform to obtain multiple word segments; Obtaining parts of speech of the multiple participles; Based on the part of speech, the sub-index value is determined.

10. The method according to claim 8, wherein The target dimension includes the picture quality dimension, and obtaining the sub-index value of the scene environment image in the target dimension includes: Determining a sub-index value of the picture quality dimension based on the evaluation index of the scene environment image; The evaluation indicators include at least one of the following: The matching degree between multiple colors contained in the scene environment image; the degree of matching between the saturation of the scene environment image and the scene type to which the target scene belongs; The stability of the scene environment image.

11. The method according to claim 8, wherein The target dimension includes the content presentation characteristic dimension, and obtaining the sub-index value of the scene environment image in the target dimension includes: Using a video understanding model to obtain video description text of the scene environment image; Analyzing the degree of matching between the video description text and the characteristic elements of the target scene to obtain a matching score; According to the matching score, a sub-index value of the content presentation characteristic dimension is determined.

12. The method according to claim 7, wherein: The image fusion strategy includes a first image display ratio; and fusing the scene environment image and the first live image according to the image fusion strategy to obtain the second live image includes: Adjusting the sizes of the first live broadcast image and the scene environment image according to the first screen display ratio; The adjusted first live image is fused with the scene environment image to obtain the second live image.

13. The method according to claim 12, wherein: The method further comprises: determining an adjustment range of the first picture display ratio according to the first picture display ratio; Within the adjustment range, adjusting the first screen display ratio according to the live broadcast duration to obtain a second screen display ratio; The scene environment image and the first live image are re-fused based on the second screen display ratio to update the second live image.

14. The method according to any one of claims 1 to 13, wherein: Generating a first live broadcast image of the digital human according to the explanation content includes: Converting the explanation content into voice information; Inputting the voice information into a digital human driving model to obtain digital population data; The digital human is rendered according to the digital population data to obtain the first live broadcast image.

15. The method according to claim 14, wherein The rendering process of the digital human according to the digital population type data to obtain the first live broadcast image includes: According to the environmental quality information, corresponding first digital human expression data is acquired; wherein the first digital human expression data includes at least one of digital human expression information and digital human movement information; The digital human is rendered according to the digital population data and the first digital human expression data to obtain the first live broadcast image.

16. The method according to claim 14, wherein The rendering process of the digital human according to the digital population data and the first digital human expression data to obtain the first live broadcast image includes: Acquiring action labels based on the explanation content through the language model; Based on the digital human type data and the digital human action data corresponding to the action tag, the digital human is rendered to obtain the first live broadcast image.

17. The method according to claim 14, wherein: The environmental quality information includes meteorological information; the method further includes: Determining the corresponding digital human background image according to the meteorological information; The digital human background image is integrated into the first live broadcast image.

18. The method according to claim 14, wherein The method further comprises: Based on the scene environment image, obtaining lighting parameters and time information; Adjust the image rendering parameters of the first live broadcast image according to the lighting parameters and the time information.

19. A live broadcast image generation device based on digital human, wherein: The device comprises: an acquisition module, configured to acquire at least one first monitoring information selected from among crowd flow information of the target scene, operation data of an interactive device provided in the target scene, a scene environment image, and environment quality information; a prompt module, configured to generate a target prompt word according to the first monitoring information; a processing module, configured to input the target prompt word into a language model to obtain explanation content output by the language model; A generating module, configured to generate a first live broadcast image of the digital human according to the explanation content; The fusion module is used to fuse the scene environment image and the first live broadcast image of the digital human to obtain a second live broadcast image of the digital human live broadcast scene.

20. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 18.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 18.

22. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 18.

Images