Augmented reality image synthesis method and device, computer equipment and storage medium

Abstract

The invention relates to the field of AR image processing, and provides an augmented reality image synthesis method and device, equipment and a storage medium, and the method comprises the steps: obtaining audio and video information to be converted into an augmented reality image, carrying out the foreground extraction of the video information, and synthesizing a target image based on the extracted target foreground and the audio information; obtaining text information to be applied to the augmented reality image; inputting the text information into a preset text processing model, and performing text processing operation on the text information in the preset text processing model to obtain target text information; performing synthesis processing on the target image and the target text information to obtain a first augmented reality image; and obtaining a to-be-fused image and a to-be-fused sound, and performing fusion processing on the to-be-fused image, the to-be-fused sound and the first augmented reality image to obtain a target augmented reality image. According to the method, intelligent explanation in a specific scene is provided for the user, the user can conveniently understand the explanation content, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of AR image processing, and in particular to a synthesis method, device, computer equipment and storage medium for augmented reality images. Background Art

[0002] At present, in the process of intelligent image explanation based on augmented reality (AR) technology, the traditional text generation model may only consider the local context, which is prone to incoherent or semantically unreasonable generation. In addition, the facial expressions and sounds of intelligent images show a single expression or a relatively simple emotional classification, which makes users prone to visual fatigue or inability to understand the content of the intelligent explanation when experiencing the intelligent image explanation based on AR technology. Therefore, how to provide users with a scene-diverse intelligent explanation, achieve a more natural and dynamic speech synthesis effect, and improve the user experience has become one of the technical problems that need to be solved urgently. Summary of the invention

[0003] The main purpose of this application is to provide a method, device, computer equipment and storage medium for synthesizing augmented reality images, aiming to synthesize augmented reality images based on user personalized characteristics, provide users with a variety of scene intelligent explanations, achieve more natural and dynamic speech synthesis effects, and improve user experience.

[0004] In a first aspect, the present application provides a method for synthesizing an augmented reality image, the method comprising the following steps: obtaining audio and video information to be converted into an augmented reality image, wherein the audio and video information includes video information and audio information; performing foreground extraction on the video information, and synthesizing a target image based on the extracted target foreground and the audio information; obtaining text information to be applied to the augmented reality image; inputting the text information into a preset text processing model, performing text processing operations on the text information in the preset text processing model, and obtaining target text information, wherein the target text information is the text information corresponding to the scene of the target image; synthesizing the target image and the target text information to obtain a first augmented reality image; obtaining an image to be fused and a sound to be fused, and fusing the image to be fused and the sound to be fused with the first augmented reality image to obtain a target augmented reality image.

[0005] In a second aspect, the present application also provides an augmented reality image synthesis device, the device comprising: an audio and video acquisition module, used to acquire audio and video information to be converted into an augmented reality image, wherein the audio and video information includes video information and audio information; a foreground extraction module, used to perform foreground extraction on the video information, and synthesize a target image based on the extracted target foreground and the audio information; a text information acquisition module, used to acquire text information to be applied to the augmented reality image; a text information processing module, used to input the text information into a preset text processing model, perform text processing operations on the text information in the preset text processing model, and obtain target text information, wherein the target text information is the text information corresponding to the scene of the target image; a first image synthesis module, used to perform synthesis processing on the target image and the target text information to obtain a first augmented reality image; a target image synthesis module, used to acquire an image to be fused and a sound to be fused, and fuse the image to be fused and the sound to be fused with the first augmented reality image to obtain a target augmented reality image.

[0006] In a third aspect, the present application further provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the method for synthesizing augmented reality images as described in any one of the first aspects is implemented.

[0007] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for synthesizing augmented reality images as described in any one of the first aspects is implemented.

[0008] The present application provides a synthesis method of augmented reality images, an apparatus, a device and a computer storage medium, the synthesis method of augmented reality images comprising: obtaining audio and video information to be converted into an augmented reality image, wherein the audio and video information comprises video information and audio information; performing foreground extraction on the video information, and synthesizing a target image based on the extracted target foreground and the audio information; obtaining text information to be applied to the augmented reality image; inputting the text information into a preset text processing model, performing text processing operation on the text information in the preset text processing model to obtain target text information, wherein the target text information is the text information corresponding to the scene of the target image; synthesizing the target image and the target text information to obtain a first augmented reality image; obtaining an image to be fused and a sound to be fused, and fusing the image to be fused and the sound to be fused with the first augmented reality image to obtain a target augmented reality image, and synthesizing a target augmented reality image with the user's personalized characteristics based on the user's personalized characteristics, providing the user with intelligent explanations in specific scenarios, achieving a more natural and dynamic speech synthesis effect, and enabling the user to better feel and understand the explanation content, thereby improving the user's experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 Shown is a flow chart of a method for synthesizing augmented reality images provided by an embodiment of the present application.

[0011] Figure 2 Shown is a flowchart of determining a target augmented reality image provided by an embodiment of the present application.

[0012] Figure 3 Shown is a flow chart of determining a target image provided by an embodiment of the present application.

[0013] Figure 4 A flowchart of determining the matching degree between color parameters and the Gaussian mixture model provided by an embodiment of the present application is shown.

[0014] Figure 5 A schematic diagram of a synthesis device for augmented reality images provided by an embodiment of the present application is shown.

[0015] Figure 6 Shown is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0018] The embodiments of the present application provide a neural network-based information processing method, apparatus, computer equipment, and storage medium.

[0019] The synthesis method of the augmented reality image can be used in a terminal or a server to achieve the purpose of synthesizing an augmented reality image based on the personalized characteristics of the user and improving the user experience. The terminal can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device; the server can be an independent server, a server cluster, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0020] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0021] See also Figure 1 , Figure 1 The flowchart of the method for synthesizing an augmented reality image provided by an embodiment of the present application is shown. Figure 1 As shown, the method for synthesizing augmented reality images includes steps S101 to S104.

[0022] S101, obtaining audio and video information to be converted into an augmented reality image, wherein the audio and video information includes video information and audio information.

[0023] Exemplarily, audio and video information to be converted into augmented reality images can be obtained based on cameras, microphones and other devices. In the process of obtaining audio and video information to be converted into augmented reality images based on cameras, microphones and other devices, attention should be paid to the quality of the obtained audio and video information, and avoiding obtaining audio and video information in noisy or dark environments.

[0024] In some embodiments, the obtaining of audio and video information to be converted into augmented reality images includes: based on a preset recording device, sequentially recording multiple pieces of audio and video information to be converted into augmented reality images, wherein the content of each piece of audio and video information is different.

[0025] The preset recording device also includes any one or more of a smart phone, a surveillance camera, a sports camera, a webcam, etc.

[0026] It should be noted that the preset video recording devices listed above are only for illustrative purposes. In actual applications, any other suitable preset video recording devices may be selected according to specific scenarios, and this application does not impose any limitation on this.

[0027] S102, extracting the foreground of the video information, and synthesizing a target image based on the extracted target foreground and the audio information.

[0028] Exemplarily, the foreground of the video information may be extracted based on the GrabCut algorithm, the foreground modeling method, the optical flow method, etc., to obtain the target foreground after foreground extraction.

[0029] Exemplarily, the target foreground and the audio information may be synthesized into a target image based on visual audio technology, an anti-reverberation model based on visual information, and the like.

[0030] It should be noted that the methods or technologies for synthesizing target images listed above are only for illustrative purposes. In actual application scenarios, any other suitable method or technology can be selected to synthesize the target image according to the specific circumstances, and this application does not impose any restrictions on this.

[0031] S103: Acquire text information to be applied to the augmented reality image.

[0032] The text information may be direct text input or obtained through voice conversion. In actual applications, appropriate text information may be selected according to specific circumstances.

[0033] S104, inputting the text information into a preset text processing model, performing text processing operations on the text information in the preset text processing model to obtain target text information, wherein the target text information is the text information corresponding to the scene of the target image.

[0034] Exemplarily, the preset text processing model may be a Transformer model.

[0035] Among them, the preset text processing model is used to perform text information synthesis, so that the content of the target text information processed by the preset text processing model is more coherent and more suitable for the application scenario.

[0036] In some embodiments, the inputting the text information into a preset text processing model, performing text processing operations on the text information in the preset text processing model, and obtaining target text information includes: obtaining historical text information and context text information corresponding to the text information; based on the historical text information and the context text information, performing text processing operations on the text information in the preset text processing model, and obtaining the target text information output by the preset text processing model.

[0037] Exemplarily, if the text information is "Today is the Winter Solstice", after the text information is input into the preset text processing model, the preset text processing model retrieves historical text information related to "Today is the Winter Solstice", which is "Eat dumplings on the Winter Solstice" and "The Winter Solstice is the day with the shortest day and the longest night in the year", and the contextual text information related to "Today is the Winter Solstice" is "I have just bought the ingredients", then the target text information obtained by the preset text processing model based on the above text information, historical text information and the contextual text information may be "Today is the Winter Solstice, the day is the shortest and the night is the longest, make dumplings early and go to bed early after dinner".

[0038] S105: synthesize the target image and the target text information to obtain a first augmented reality image.

[0039] Exemplarily, the target image and the target text information may be synthesized based on deep learning technology, a neural network model, etc. to obtain a first augmented reality image.

[0040] It should be noted that the above-mentioned technology of synthesizing the target image and the target text information is only used for exemplary description, and this application does not impose any limitation on this.

[0041] S106, obtaining the image to be fused and the sound to be fused, and fusing the image to be fused and the sound to be fused with the first augmented reality image to obtain a target augmented reality image.

[0042] The image to be fused may be the user's facial expression, gesture, etc. The sound to be fused may be the user's personalized voice, for example, the sound to be fused may be the voice of a young person, a child, an old person, etc.

[0043] Among them, by fusing the image to be fused and the sound to be fused with the first augmented reality image, the target augmented reality image obtained can more accurately reflect the user's personalized characteristics, shorten the distance between the user and the target augmented reality image in the AR scene, and improve the user's experience.

[0044] It should be noted that the above-listed synthesis technologies and methods of obtaining text information are only for illustrative purposes. In actual application scenarios, any suitable synthesis technology and method of obtaining text information can be selected according to the specific circumstances, and this application does not impose any other restrictions.

[0045] The embodiment of the present application provides a method for synthesizing an augmented reality image, the method comprising: obtaining audio and video information to be converted into an augmented reality image, wherein the audio and video information comprises video information and audio information; performing foreground extraction on the video information, and synthesizing a target image based on the extracted target foreground and the audio information; obtaining text information to be applied to the augmented reality image; inputting the text information into a preset text processing model, performing text processing operations on the text information in the preset text processing model, and obtaining target text information, wherein the target text information is the text information corresponding to the scene of the target image; synthesizing the target image and the target text information to obtain a first augmented reality image; obtaining an image to be fused and a sound to be fused, and merging the image to be fused and the sound to be fused with the first augmented reality image to obtain a target augmented reality image, wherein the target augmented image obtained by merging the image to be fused and the sound to be fused with the first augmented reality image adds the personalized features of the user, can provide the user with personalized intelligent explanations, facilitate the user to better understand the content of the explanations, and improve the user's experience.

[0046] See also Figure 2 , Figure 2 The flowchart of determining the target augmented reality image provided by an embodiment of the present application is shown. Figure 2 As shown, the method for determining a target augmented reality image includes steps S201 to S205.

[0047] S201, determining image features corresponding to the image to be fused.

[0048] Among them, image features can be: gender of the person, weather, location, scene, etc.

[0049] S202, adding the image feature to the image to be fused to obtain a target image to be fused corresponding to the image to be fused.

[0050] For example, if the image features are little girl, summer, museum, and visiting cultural relics, then the target image to be fused based on the above image features is a little girl wearing a T-shirt and visiting cultural relics in a museum.

[0051] S203: Determine the emotional features corresponding to the sound to be fused.

[0052] Among them, emotional characteristics can be: happy, excited, sad, painful, etc.

[0053] S204, based on the emotion synthesis technology, adding the emotion feature to the sound to be fused, to obtain the target sound to be fused corresponding to the sound to be fused.

[0054] For example, if the emotional feature is happiness, the target sound to be fused will be a lighter sound.

[0055] S205: Perform fusion processing on the first augmented reality image based on the target image to be fused and the target sound to be fused, so as to obtain the target augmented reality image.

[0056] For example, if the target image to be fused is: a little girl in a T-shirt visiting cultural relics in a museum. The target image to be fused is: a brisk voice, then the target augmented reality image obtained is: a little girl in a T-shirt visiting cultural relics in a museum. The little girl's explanation of the cultural relics is a child's voice, and the voice is relatively brisk and closer to the little girl's own voice, which improves the little girl's explanation experience.

[0057] The present application provides a method for determining a target augmented reality image, in which the image features and emotional features corresponding to the image to be fused are determined, and the image features and emotional features are added to the image to be fused and the sound to be fused respectively, and the target image to be fused and the target sound to be fused are fused with the first augmented reality image to obtain the target augmented reality image, and specific image features and emotional features are added to the first augmented image, so that the target augmented image obtained based on the first augmented image is more vivid and fits the personality of the user, and different users can have their own personalized expressions, which improves the user's experience. The applicable range of users is expanded.

[0058] See also Figure 3 , Figure 3 The flowchart of determining the target image provided by an embodiment of the present application is shown. Figure 3 As shown, the method for determining the target image includes steps S301 to S304.

[0059] S301, establishing a Gaussian mixture model for each pixel in the video information, wherein the Gaussian mixture model includes at least one Gaussian function.

[0060] Specifically, before establishing the Gaussian mixture model, the Gaussian mixture model is initialized, wherein initializing the Gaussian mixture model includes: initializing the mean, standard deviation and weight of each Gaussian model.

[0061] S302, obtaining color parameters of each pixel point in the current video frame, and matching the color parameters with each Gaussian function to obtain a matching degree between the color parameters and the Gaussian mixture model.

[0062] The color parameters include: phase, saturation, brightness, RGB value, etc. For each pixel, the matching degree between its color parameters and each Gaussian distribution is calculated. The matching degree between its color parameters and each Gaussian distribution can be calculated by calculating the Mahalanobis distance (MD) between the color parameters of the pixel and the mean of the Gaussian distribution.

[0063] For example, for each new pixel value, it is compared with the current K Gaussian models to find a distribution model that matches the new pixel value. The matching degree can be that the deviation between the pixel value and the model mean is within 2.5 standard deviations.

[0064] S303: Determine the background probability that the pixel point belongs to the video background according to the matching degree.

[0065] Specifically, if a pixel successfully matches a Gaussian distribution, the pixel is considered to belong to the background, otherwise it belongs to the foreground. The background probability can be determined by the weight of the matched Gaussian distribution. The larger the weight, the higher the background probability.

[0066] S304: Determine a target foreground of the video information according to the background probability.

[0067] Specifically, a threshold can be set to determine whether the pixel belongs to the background according to the matching degree between the pixel and the Gaussian distribution and the background probability. If the background probability is higher than the threshold, it is determined to be the background; otherwise, it is determined to be the foreground.

[0068] Exemplarily, the ratio of the weight to the variance of the Gaussian model can be arranged in descending order, and the first B Gaussian distributions that meet a specific threshold T can be selected as the model for judging whether the current pixel belongs to the video background. For example, if a new pixel value obeys one of the B Gaussian models, the pixel is considered to belong to the background. On the contrary, if the Gaussian distribution corresponding to the pixel does not meet the threshold T, the pixel is the foreground. The specific threshold T can be a suitable value such as 0.7, 0.8, etc. The above-mentioned method based on the matching degree of color parameters and Gaussian mixture models improves the accuracy of determining the target foreground.

[0069] See also Figure 4 , Figure 4 The flowchart of determining the matching degree between the color parameters and the Gaussian mixture model provided by an embodiment of the present application is shown. Figure 4 As shown, the method for determining the matching degree between the color parameter and the Gaussian mixture model includes steps S401 to S404.

[0070] S401, determining the mean, variance and weight of at least one Gaussian function according to the historical color parameters of each pixel point, and obtaining a Gaussian mixture model corresponding to the pixel point.

[0071] Exemplarily, the mean, variance and weight of at least one of the Gaussian functions may be determined based on the EM algorithm to obtain a Gaussian mixture model corresponding to the pixel point.

[0072] S402, determining a Gaussian function belonging to the background Gaussian function according to the weight and the variance.

[0073] Specifically, a weight threshold and a variance threshold are set. The weight threshold is used to filter out Gaussian distributions with higher weights, and the variance threshold is used to filter out Gaussian distributions with smaller variances. Based on the set weight threshold and variance threshold, the Gaussian function distribution with the highest weight and the smallest variance is filtered out as the Gaussian function.

[0074] In some embodiments, the obtaining of each Gaussian function determined to be a background Gaussian function according to the weight and the variance includes: determining a Gaussian function whose ratio of weight to variance is greater than a preset threshold as the background Gaussian function.

[0075] S403, matching the color parameter with each of the background Gaussian functions to obtain a matching degree between the color parameter and the Gaussian mixture model.

[0076] The matching degree between the color parameter and the Gaussian mixture model can be achieved based on the Mahalanobis distance formula.

[0077] The formula for Mahalanobis distance is: x represents the color parameter, μ represents the mean of the Gaussian distribution, and Σ represents the covariance matrix (for a univariate Gaussian distribution, it is the inverse of the variance).

[0078] The embodiment of the present application provides a method for determining the matching degree between color parameters and the Gaussian mixture model, in which the Gaussian mixture model can flexibly adjust the number of Gaussian distributions to adapt to AR scenes of different complexities. By calculating the matching degree between pixels and Gaussian distributions, the Gaussian mixture model can more accurately distinguish between foreground and background, thereby improving the accuracy of the subsequent matching degree between color parameters and the Gaussian mixture model.

[0079] See also Figure 5 , Figure 5 Shown is a schematic diagram of an augmented reality image synthesis device provided by an embodiment of the present application. The augmented reality image synthesis device can be configured in a server or a terminal to execute the aforementioned augmented reality image synthesis method.

[0080] like Figure 5 As shown, the augmented reality image synthesis device 500 includes: an audio and video acquisition module 501, a foreground extraction module 502, a text information acquisition module 503, a text information processing module 504, a first image synthesis module 505, and a target image synthesis module 506.

[0081] The audio and video acquisition module 501 is used to acquire audio and video information to be converted into augmented reality images, wherein the audio and video information includes video information and audio information.

[0082] The foreground extraction module 502 is used to extract the foreground of the video information and synthesize the target image based on the extracted target foreground and the audio information.

[0083] The text information acquisition module 503 is used to acquire text information to be applied to the augmented reality image.

[0084] The text information processing module 504 is used to input the text information into a preset text processing model, perform text processing operations on the text information in the preset text processing model, and obtain target text information, wherein the target text information is the text information corresponding to the scene of the target image.

[0085] The first image synthesis module 505 is used to synthesize the target image and the target text information to obtain a first augmented reality image.

[0086] The target image synthesis module 506 is used to obtain the image to be fused and the sound to be fused, and fuse the image to be fused and the sound to be fused with the first augmented reality image to obtain a target augmented reality image.

[0087] In some embodiments, the audio and video acquisition module 501 is specifically used to: acquire audio and video information to be converted into an augmented reality image, wherein the audio and video information includes video information and audio information;

[0088] In some embodiments, the audio and video acquisition module 501 is further specifically used to: based on a preset recording device, sequentially record multiple audio and video information to be converted into augmented reality images, wherein the content of each audio and video information is different.

[0089] In some embodiments, the foreground extraction module 502 is specifically used to: extract the foreground of the video information, and synthesize the target image based on the extracted target foreground and the audio information.

[0090] In some embodiments, the foreground extraction module 502 is further specifically used to: establish a Gaussian mixture model for each pixel in the video information, wherein the Gaussian mixture model includes at least one Gaussian function; obtain color parameters of each pixel in the current video frame, and match the color parameters with each Gaussian function to obtain a matching degree between the color parameters and the Gaussian mixture model; determine the background probability that the pixel belongs to the video background based on the matching degree; and determine the target foreground of the video information based on the background probability.

[0091] In some embodiments, the foreground extraction module 502 is further specifically used to: determine the mean, variance and weight of at least one of the Gaussian functions based on the historical color parameters of each of the pixel points to obtain a Gaussian mixture model corresponding to the pixel point; determine the Gaussian function belonging to the background Gaussian function based on the weight and variance; match the color parameters with each of the background Gaussian functions to obtain a matching degree between the color parameters and the Gaussian mixture model.

[0092] In some embodiments, the foreground extraction module 502 is further configured to: determine a Gaussian function whose weight to variance ratio is greater than a preset threshold as the background Gaussian function.

[0093] In some embodiments, the text information acquisition module 503 is specifically used to: acquire text information to be applied to the augmented reality image.

[0094] In some embodiments, the text information processing module 504 is specifically used to: input the text information into a preset text processing model, perform text processing operations on the text information in the preset text processing model, and obtain target text information, wherein the target text information is the text information corresponding to the scene of the target image.

[0095] In some embodiments, the text information processing module 504 is further specifically used to: obtain historical text information and context text information corresponding to the text information; based on the historical text information and the context text information, perform text processing operations on the text information in the preset text processing model to obtain the target text information output by the preset text processing model.

[0096] In some embodiments, the first image synthesis module 505 is specifically used to synthesize the target image and the target text information to obtain a first augmented reality image.

[0097] In some embodiments, the target image synthesis module 506 is specifically used to obtain the image to be fused and the sound to be fused, and fuse the image to be fused and the sound to be fused with the first augmented reality image to obtain the target augmented reality image.

[0098] In some embodiments, the target image synthesis module 506 is further specifically used to: determine the image features corresponding to the image to be fused; add the image features to the image to be fused to obtain the target image to be fused corresponding to the image to be fused; determine the emotional features corresponding to the sound to be fused; based on the emotional synthesis technology, add the emotional features to the sound to be fused to obtain the target sound to be fused corresponding to the sound to be fused; perform fusion processing based on the target image to be fused and the target sound to be fused with the first augmented reality image to obtain the target augmented reality image.

[0099] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0100] The method and apparatus of the present application can be used in many general or special computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0101] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in Figure 6 Runs on the computer device shown.

[0102] See also Figure 6 , Figure 6 The figure shows a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server or a terminal.

[0103] like Figure 6As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a storage medium and an internal memory.

[0104] The storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any information processing method based on a neural network.

[0105] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0106] The internal memory provides an environment for the operation of the computer program in the storage medium. When the computer program is executed by the processor, the processor can execute any information processing method based on a neural network.

[0107] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0108] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0109] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0110] Acquire audio and video information to be converted into an augmented reality image, wherein the audio and video information includes video information and audio information; perform foreground extraction on the video information, and synthesize a target image based on the extracted target foreground and the audio information; acquire text information to be applied to the augmented reality image; input the text information into a preset text processing model, and perform text processing operations on the text information in the preset text processing model to obtain target text information, wherein the target text information is the text information corresponding to the scene of the target image; perform synthesis processing on the target image and the target text information to obtain a first augmented reality image; acquire an image to be fused and a sound to be fused, and fuse the image to be fused and the sound to be fused with the first augmented reality image to obtain a target augmented reality image.

[0111] In one embodiment, when the processor acquires the image to be fused and the sound to be fused, and fuses the image to be fused and the sound to be fused with the first augmented reality image to obtain the target augmented reality image, it is used to implement:

[0112] Determine the image features corresponding to the image to be fused; add the image features to the image to be fused to obtain a target image to be fused corresponding to the image to be fused; determine the emotional features corresponding to the sound to be fused; based on the emotional synthesis technology, add the emotional features to the sound to be fused to obtain a target sound to be fused corresponding to the sound to be fused; perform fusion processing based on the target image to be fused and the target sound to be fused with the first augmented reality image to obtain the target augmented reality image.

[0113] In one embodiment, when the processor extracts the foreground of the video information and synthesizes the target image based on the extracted target foreground and the audio information, it is used to implement:

[0114] A Gaussian mixture model is established for each pixel in the video information, wherein the Gaussian mixture model includes at least one Gaussian function; color parameters of each pixel in the current video frame are obtained, and the color parameters are matched with each Gaussian function to obtain a matching degree between the color parameters and the Gaussian mixture model; based on the matching degree, a background probability that the pixel belongs to the video background is determined; and a target foreground of the video information is determined based on the background probability.

[0115] In one embodiment, the processor, when obtaining the color parameters of each pixel point in the current video frame and matching the color parameters with each Gaussian function to obtain the matching degree between the color parameters and the Gaussian mixture model, is used to implement:

[0116] According to the historical color parameters of each of the pixel points, the mean, variance and weight of at least one of the Gaussian functions are determined to obtain a Gaussian mixture model corresponding to the pixel point; the Gaussian function belonging to the background Gaussian function is determined according to the weight and variance; the color parameters are matched with each of the background Gaussian functions to obtain a matching degree between the color parameters and the Gaussian mixture model.

[0117] In one embodiment, when the processor inputs the text information into a preset text processing model and performs a text processing operation on the text information in the preset text processing model to obtain target text information, it is used to implement:

[0118] Obtain historical text information and context text information corresponding to the text information; based on the historical text information and the context text information, perform text processing operations on the text information in the preset text processing model to obtain target text information output by the preset text processing model.

[0119] In one embodiment, when acquiring the audio and video information to be converted into an augmented reality image, the processor is used to implement:

[0120] Based on a preset recording device, multiple audio and video information to be converted into augmented reality images are recorded in sequence, wherein the content of each audio and video information is different.

[0121] In one embodiment, the processor is further configured to: determine a Gaussian function whose weight to variance ratio is greater than a preset threshold as the background Gaussian function.

[0122] It should be noted that those skilled in the art can clearly understand that, for the sake of convenience and brevity of description, the specific working process of the above-described device for synthesizing augmented reality images can refer to the corresponding process in the aforementioned embodiment of the method for synthesizing augmented reality images, and will not be repeated here.

[0123] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the information processing method based on a neural network in the present application.

[0124] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0125] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0126] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0127] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for synthesizing an augmented reality image, characterized in that: include: Acquire audio and video information to be converted into augmented reality images, wherein the audio and video information includes video information and audio information; Extracting a foreground from the video information, and synthesizing a target image based on the extracted target foreground and the audio information; Acquire text information to be applied to the augmented reality image; Inputting the text information into a preset text processing model, performing a text processing operation on the text information in the preset text processing model to obtain target text information, wherein the target text information is the text information corresponding to the scene of the target image; Performing synthesis processing on the target image and the target text information to obtain a first augmented reality image; The image to be fused and the sound to be fused are obtained, and the image to be fused and the sound to be fused are fused with the first augmented reality image to obtain a target augmented reality image.

2. The method for synthesizing augmented reality images according to claim 1, characterized in that: The acquiring the image to be fused and the sound to be fused, and fusing the image to be fused and the sound to be fused with the first augmented reality image to obtain a target augmented reality image, includes: Determining image features corresponding to the image to be fused; Adding the image feature to the image to be fused to obtain a target image to be fused corresponding to the image to be fused; Determining the emotional features corresponding to the sound to be fused; Based on the emotion synthesis technology, the emotion feature is added to the sound to be fused to obtain the target sound to be fused corresponding to the sound to be fused; The target augmented reality image is obtained by performing fusion processing on the target image to be fused and the target sound to be fused with the first augmented reality image.

3. The method for synthesizing augmented reality images according to claim 1, characterized in that: The extracting the foreground of the video information and synthesizing the target image based on the extracted target foreground and the audio information includes: Establishing a Gaussian mixture model for each pixel in the video information, wherein the Gaussian mixture model includes at least one Gaussian function; Acquire color parameters of each pixel point in the current video frame, and match the color parameters with each Gaussian function to obtain a matching degree between the color parameters and the Gaussian mixture model; Determining the background probability that the pixel point belongs to the video background according to the matching degree; A target foreground of the video information is determined according to the background probability.

4. The method for synthesizing augmented reality images according to claim 3, characterized in that: The step of obtaining the color parameters of each pixel point in the current video frame and matching the color parameters with each Gaussian function to obtain the matching degree between the color parameters and the Gaussian mixture model includes: Determine the mean, variance and weight of at least one Gaussian function according to the historical color parameters of each pixel point, and obtain a Gaussian mixture model corresponding to the pixel point; Determine a Gaussian function belonging to the background Gaussian function according to the weight and the variance; The color parameters are matched with each of the background Gaussian functions to obtain a matching degree between the color parameters and the Gaussian mixture model.

5. The method for synthesizing augmented reality images according to claim 4, characterized in that: The obtaining of each Gaussian function determined to belong to the background Gaussian function according to the weight and the variance comprises: A Gaussian function whose ratio of weight to variance is greater than a preset threshold is determined as the background Gaussian function.

6. The method for synthesizing augmented reality images according to claim 1, characterized in that: The step of inputting the text information into a preset text processing model, performing a text processing operation on the text information in the preset text processing model, and obtaining target text information includes: Obtaining historical text information and context text information corresponding to the text information; Based on the historical text information and the context text information, a text processing operation is performed on the text information in the preset text processing model to obtain target text information output by the preset text processing model.

7. The method for synthesizing augmented reality images according to any one of claims 1 to 6, characterized in that: The step of obtaining audio and video information to be converted into augmented reality images includes: Based on a preset recording device, multiple audio and video information to be converted into augmented reality images are recorded in sequence, wherein the content of each audio and video information is different.

8. A synthesis device for augmented reality images, characterized in that: include: An audio and video acquisition module, used to acquire audio and video information to be converted into augmented reality images, wherein the audio and video information includes video information and audio information; A foreground extraction module, used to extract the foreground of the video information, and synthesize a target image based on the extracted target foreground and the audio information; A text information acquisition module, used to acquire text information to be applied to the augmented reality image; A text information processing module, used for inputting the text information into a preset text processing model, performing a text processing operation on the text information in the preset text processing model, and obtaining target text information, wherein the target text information is the text information corresponding to the scene of the target image; A first image synthesis module, used for synthesizing the target image and the target text information to obtain a first augmented reality image; The target image synthesis module is used to obtain the image to be fused and the sound to be fused, and fuse the image to be fused and the sound to be fused with the first augmented reality image to obtain a target augmented reality image.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the method for synthesizing augmented reality images as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method for synthesizing augmented reality images as described in any one of claims 1 to 7.

Images