Image processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment and a storage medium. The method comprises the steps that audio data and target part data corresponding to a target object are acquired; determining first data to be fused corresponding to the audio data; determining second to-be-fused data corresponding to the target part data; and determining target fusion data based on the first to-be-fused data and the second to-be-fused data, so as to drive a target virtual object to be displayed based on the target fusion data. According to the technical scheme provided by the embodiment of the invention, the technical effects that the audio information and the part data of the target object are processed online, the target fusion data are determined, and the three-dimensional virtual object is driven to be displayed based on the target fusion data are achieved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular, to an image processing method, device, electronic device, and storage medium. Background Art

[0002] With the development of computer technology, more and more application software have emerged, and users have more and more functional requirements for application software. Correspondingly, the demand for interaction with terminal devices also needs to be improved.

[0003] At present, the expression driving of virtual objects is mainly based on the character's lip shape information. When based on the lip shape information, there is inaccurate character pronunciation, which leads to inaccurate expression driving. Alternatively, by recording lip shape animation and then driving the expression, online processing cannot be achieved at this time, that is, there is a long lag and delay.

[0004] In other words, the existing expression-driven virtual objects cannot be realized in real time and have a long lag. Summary of the invention

[0005] The present disclosure provides an image processing method, device, electronic device and storage medium to achieve real-time driving of a virtual object to simulate the lip shape and expression corresponding to a target object, so as to achieve the effects of interactive realism and driving synchronization.

[0006] In a first aspect, an embodiment of the present disclosure provides an image processing method, the method comprising:

[0007] Acquire audio data and target part data corresponding to the target object; wherein the target part data corresponds to the position information and state information of the preset acquisition part;

[0008] Determining first data to be fused corresponding to the audio data; and,

[0009] Determining second data to be fused corresponding to the target part data;

[0010] Based on the first data to be fused and the second data to be fused, target fusion data is determined, so as to drive a target virtual object to be displayed based on the target fusion data.

[0011] In a second aspect, an embodiment of the present disclosure further provides an image processing device, the device comprising:

[0012] A data acquisition module, used to acquire audio data and target part data corresponding to the target object; wherein the target part data corresponds to the position information and state information of the preset acquisition part; a first data determination module, used to determine the first data to be fused corresponding to the audio data; and

[0013] A second data determination module, used to determine second data to be fused corresponding to the target part data;

[0014] The target fusion data determination module is used to determine target fusion data based on the first data to be fused and the second data to be fused, so as to drive the target virtual object to be displayed based on the target fusion data.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of the embodiments of the present disclosure.

[0019] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the image processing method as described in any one of the embodiments of the present disclosure.

[0020] The technical solution of the disclosed embodiment can collect audio data and target part data of the target object in real time, determine lip shape data by analyzing and processing the audio data, and determine facial expression data by analyzing and processing the target part data, and further fuse the lip shape data and facial expression data to obtain target fusion data for driving the target virtual object. This solves the problem in the prior art that when driving the virtual object by expression, it is impossible to achieve real-time performance and thus there is a long lag, and realizes real-time analysis and processing of the collected data, thereby achieving the technical effect of synchronously driving the target virtual object. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0023] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0024] Figure 3is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0025] Figure 4 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0033] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0034] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0035] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0036] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0037] Before introducing the technical solution, an exemplary description of the application scenario can be given. The technical solution provided by the embodiment of the present disclosure can be applied to any scenario in which a virtual object is driven to be displayed based on the voice and part data of the target object. For example, it can be applied to a live broadcast scene, a video conference scene, a video call scene, and a special effect display scene. The specific application scenario is not specifically limited in the solution provided by the embodiment of the present disclosure. Among them, the live broadcast scene can be a scene of live broadcasting of selling items. For example, if it is a live broadcast scene, for a live broadcast user, if he does not want to appear on camera, he can create a virtual object, and obtain the audio information and target part data corresponding to the live broadcast object in real time during the live broadcast of the live broadcast user. By analyzing and processing the above data, the effect of driving the lip shape and part display information of the virtual object to be consistent with the live broadcast user is achieved. It can also be a special effect scene, that is, the lip shape data and target part data of the target object are collected in real time to drive the virtual object to display.

[0038] It should be noted that the above has briefly described various applicable scenarios, which will not be elaborated in detail here. Among them, the device for executing the image method provided by the embodiment of the present disclosure can be integrated in an application software that supports video image processing functions, and the software can be installed in an electronic device. Optionally, the electronic device can be a mobile terminal or a PC. The application software can be a type of software for image / video processing, and its specific application software will not be described one by one here, as long as image / video processing can be achieved.

[0039] Figure 1 It is a flow chart of an image processing method provided by an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to any situation where a virtual object needs to be driven for anthropomorphic display. The method can be executed by an image processing device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc.

[0040] like Figure 1 As shown, the method includes:

[0041] S110: Acquire audio data and target part data corresponding to the target object.

[0042] Usually, when driving a three-dimensional virtual object for display, a corresponding reference is required, that is, which object is used as a prototype for display, and the prototype can be used as a target object. For example, if the lip shape and expression displayed in real time by the virtual object correspond to user A, the audio data and target part data of user A can be obtained, and user A can be used as the target object. During the target object's speech, the corresponding audio information can be collected in real time and used as audio data. Correspondingly, during the target object's speech, there are usually different expressions. In order to achieve the effect of realistic display of the virtual object, the target part data of different parts of the target object's facial image can be obtained during the target object's speech. In other words, the target part data is the data corresponding to different facial features on the target object's face. Of course, the target part data can be data corresponding to only at least one of the eyebrow part, nose part, eye part, mouth part and ear part.

[0043] Specifically, audio data corresponding to the target object can be collected in real time based on the deployed microphone array, and target part data of the target part can be collected based on the facial image collection device.

[0044] It should also be noted that the target part corresponding to the target part data can be pre-set or determined based on the trigger selection of the target object. The specific selection method of the target part is not limited in the embodiment of the present disclosure and can be selected according to actual needs.

[0045] The target part data corresponds to the position information and state information of the preset collection part; wherein the preset collection part may be a pre-set part, and optionally, the preset collection part may be a nose, eyebrow, mouth, eye and other parts of the face. The position information may be the specific position of each part on the face, and the state may be an open or closed state.

[0046] Optionally, the acquiring of audio data and target part data corresponding to the target object includes: in response to a virtual object driving operation, collecting audio information corresponding to the target object based on an audio collection device; and collecting target part data corresponding to the target object based on a facial image collection device; wherein the part corresponding to the target part data includes at least one part of the five senses.

[0047] It can be understood that whether to drive the virtual object is determined based on the actual needs of the target object. When an operation that triggers the driving of the virtual object is detected, the audio data and target part data of the target object can be collected. The audio information of the target object can be collected based on devices such as smart speakers, smart robots, chat robots, microphone arrays, etc. The output of the audio data is output based on a wired or wireless method. The facial image acquisition device can be a camera device, which can be a depth camera, etc. If the target part is consistent with one or more pre-set parts, after the facial image is collected, the feature data of one or more parts can be extracted as the target part data. If the target part is not set, the entire facial image data collected can be used as the target part data.

[0048] S120: Determine first data to be fused corresponding to the audio data.

[0049] It should be noted that the audio data is the most original data collected. In order to drive the facial image of the virtual object to be consistent with the target object, the audio data can be further analyzed and processed to obtain the first data to be fused. In other words, the first data to be fused is the data corresponding to the audio data after analysis and processing.

[0050] In this embodiment, determining the first data to be fused corresponding to the audio data includes: determining a plurality of characters corresponding to the audio data, and determining pronunciation information corresponding to the plurality of characters; and determining the first data to be fused in the mouth part based on the pronunciation information.

[0051] Among them, the audio data is the most original voice information collected. The audio data can be analyzed and processed by the speech-to-text module to obtain text information corresponding to the audio data. At this time, the text information and sentence semantics can be analyzed and processed to determine the pronunciation information corresponding to each text. This method avoids the problem of inaccurate determination of the first data to be fused due to inaccurate pronunciation of the target object. The lip shape data can be determined based on the pronunciation information, and the lip shape data is used as the first data to be fused.

[0052] It should also be noted that the virtual object is usually a three-dimensional virtual object. In order to make the mouth display information of the virtual object consistent with the mouth display information of the target object, the face can be divided into multiple grids in advance, that is, the face is composed of multiple face patches. The grid data corresponding to each grid can be determined based on the audio data. In other words, the first data to be fused is the grid point data converted after the voice information is processed.

[0053] S130: Determine second data to be fused corresponding to the target part data.

[0054] The target part data is data that matches the preset target part. The second data to be fused is data obtained after processing the target part data.

[0055] In this embodiment, determining the second data to be fused corresponding to the target part data includes: determining grid point data of a plurality of grids corresponding to the target part based on the target part data; and using the grid point data as the second data to be fused.

[0056] Based on S120, it can be known that the virtual object is a three-dimensional virtual object. In order to achieve a three-dimensional and realistic display effect, the facial area can be divided into multiple grid areas. At this time, the area covered by the grid area is consistent with the facial area of ​​the three-dimensional virtual object. Based on the collected facial image data, the grid point data corresponding to the grid point of each grid can be determined. The grid point data determined at this time is used as the second data to be fused.

[0057] It should also be noted that the target part data corresponds to at least one facial part, and the integration of facial parts can correspond to the facial expression of the target object. Accordingly, the target part data corresponds to the facial expression of the target object. That is, the target part data is the facial expression data of the target object during the speaking process.

[0058] S140: Determine target fusion data based on the first data to be fused and the second data to be fused, so as to drive a target virtual object to be displayed based on the target fusion data.

[0059] The target fusion data is the data obtained after the fusion processing of the first data to be fused and the second data to be fused. At this time, the target fusion data includes the processed lip shape data and the processed facial expression data.

[0060] Specifically, the first data to be fused and the second data to be fused may be fused according to a preset rule to obtain target fused data corresponding to the mouth shape and facial expression of the target object. After obtaining the target fused data, the target virtual object may be driven to be displayed, and at this time, the mouth shape and facial expression of the target virtual object match the mouth shape and facial expression of the target object.

[0061] The technical solution of the disclosed embodiment can collect audio data and target part data of the target object in real time, determine lip shape data by analyzing and processing the audio data, and determine facial expression data by analyzing and processing the target part data, and further fuse the lip shape data and facial expression data to obtain target fusion data for driving the target virtual object. This solves the problem in the prior art that when driving the virtual object by expression, it is impossible to achieve real-time performance and thus there is a long lag, and realizes real-time analysis and processing of the collected data, thereby achieving the technical effect of synchronously driving the target virtual object.

[0062] Figure 2 A flow chart of an image processing method provided in the second embodiment of the present disclosure is provided. On the basis of the above-mentioned embodiment, further explanation can be given on how to determine the target fusion data. The specific implementation method can be found in the detailed description of this embodiment, wherein the technical terms that are the same as or corresponding to the above-mentioned embodiment are not repeated here.

[0063] like Figure 2 The method shown comprises:

[0064] S210: Acquire audio data and target part data corresponding to the target object.

[0065] Exemplarily, in a live broadcast scenario, during the live broadcast of the target object, audio data and facial data corresponding to the target object, that is, audio data and target part data, may be collected in real time.

[0066] S220: Determine first data to be merged corresponding to the audio data.

[0067] By analyzing and processing the audio data, the lip shape data corresponding to the audio information is determined, that is, the first data to be fused. The first data to be fused can be composed of 52 blendshape (single grid point) data. That is, after analyzing and processing the audio data, the lip shape data and the rough expression data are fitted.

[0068] S230: Determine second data to be fused corresponding to the target part data.

[0069] In the process of determining the first data to be fused, the target part data can also be analyzed and processed to obtain 52 blendshape data corresponding to the facial expression. That is, the data at this time is the facial expression data determined directly based on the collected target part data.

[0070] S240, determining the maximum value between the first data to be fused and the second data to be fused corresponding to the same grid point, and taking the maximum value as the target grid point data of the corresponding grid point.

[0071] It can be understood that, for each grid point, the data processing method is the same, and here the grid data of one grid point is determined as an example for explanation.

[0072] It should also be noted that the number of grid points corresponding to the first data to be fused and the second data to be fused determined based on the audio information is the same, but the emphasis of the data is different.

[0073] Specifically, for the data corresponding to the same grid point, the maximum value of the first fused data to be processed and the second fused data to be processed is determined, and the maximum value is used as the target grid data of the corresponding grid point.

[0074] Exemplarily, based on the above steps, two sets of data are obtained, the first set of first data to be fused is lip shape blendshape data, and the other set of second data to be fused is expression blendshape data. For each blendshape data, the maximum value is selected as the target fusion data corresponding to the corresponding blendshape data.

[0075] S250: Determine the target fusion data according to the target grid point data of at least part of the grid points.

[0076] That is, the target fusion data includes the target grid point data of each grid point. Based on the target fusion data, the target virtual object can be driven to display to achieve the effect of real-time drive display. The specific display can refer to the description of the above embodiment. At least some of the grid points can be all of the grid points or some of the grid points.

[0077] In practical applications, if the virtual object display is driven based on the target fusion data at this time, the facial expressions and mouth shape of the target virtual object may jump or become unstable during the speaking process. The target grid point data of each target grid point can be processed based on the fusion curve to obtain the target fusion data.

[0078] The fusion curve may correspond to different preset expressions. The fusion curve may include fusion values ​​corresponding to different preset expressions at each grid point. The fusion curve may also obtain several video frames before the current frame, and update the fusion value corresponding to each grid point according to the grid point data of the previous video frames.

[0079] That is to say, after obtaining the target fusion data, the target fusion data can be slightly adjusted based on the fusion curve, ensuring the continuity and stability of the expression changes of the virtual object in continuous video frames, so as to more accurately express the real-time status of the target virtual object.

[0080] For example, in order to avoid data jumps and instability, after determining the target fusion data, the target fusion data is changed in a curve, that is, the target fusion data is fine-tuned in combination with the historical target fusion data corresponding to the previous video frame to update the target fusion data. When the target virtual object is displayed based on the target fusion data obtained at this time, the continuity and stability of the expression are guaranteed, and the real-time status of the target virtual object can be more accurately expressed.

[0081] The technical solution of the disclosed embodiment can collect audio data and target part data of the target object in real time, determine lip shape data by analyzing and processing the audio data, and determine facial expression data by analyzing and processing the target part data, and further fuse the lip shape data and facial expression data to obtain target fusion data for driving the target virtual object. This solves the problem in the prior art that when driving the virtual object by expression, it is impossible to achieve real-time performance and thus there is a long lag, and realizes real-time analysis and processing of the collected data, thereby achieving the technical effect of synchronously driving the target virtual object.

[0082] Figure 3 is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure, such as Figure 3 As shown, the device includes: a data acquisition module 310, a first data determination module 320, a second data determination module 330 and a target fusion data determination module 340.

[0083] A data acquisition module 310 is used to acquire audio data and target part data corresponding to a target object, wherein the target part data corresponds to position information and status information of a preset acquisition part; a first data determination module 320 is used to determine first data to be fused corresponding to the audio data; and a second data determination module 330 is used to determine second data to be fused corresponding to the target part data; a target fusion data determination module 340 is used to determine target fusion data based on the first data to be fused and the second data to be fused, so as to drive the target virtual object to be displayed based on the target fusion data.

[0084] On the basis of the above technical solution, the data acquisition module includes:

[0085] An audio data acquisition unit is used to collect audio information corresponding to the target object based on an audio acquisition device in response to a virtual object driving operation; and a part data acquisition unit is used to collect target part data corresponding to the target object based on a facial image acquisition device; wherein the part corresponding to the target part data includes at least one part of the five senses.

[0086] Based on the above technical solutions, the first data determination module includes:

[0087] The pronunciation information determination unit is used to determine multiple text information corresponding to the audio data and determine the pronunciation information corresponding to the multiple text information characters; the first fusion unit is used to determine the first data to be fused of the mouth part based on the pronunciation information.

[0088] On the basis of the above technical solutions, the second data determination module includes:

[0089] A grid point data determination unit is used to determine the grid point data of multiple grids corresponding to the target part based on the target part data; and a second fusion unit is used to use the grid point data as the second data to be fused.

[0090] On the basis of the above technical solutions, the target fusion data determination module includes:

[0091] The grid point data determination unit is used to determine the maximum value of the first data to be fused and the second data to be fused corresponding to the same grid point, and use the maximum value as the target grid point data of the corresponding grid point; the target fusion unit is used to determine the target fusion data based on the target grid point data of at least some grid points.

[0092] On the basis of the above technical solutions, the target fusion unit includes:

[0093] The superposition fusion subunit is used to process the target grid point data according to a preset fusion curve to determine the to-be-superimposed fusion data corresponding to the target grid point; the target fusion subunit is used to update the target fusion data based on the to-be-fused superposition data.

[0094] On the basis of the above technical solutions, the fusion curve corresponds to a preset expression.

[0095] Based on the above technical solutions, the device also includes: a driving module, which is used to update the facial expression and lip shape of the target virtual object based on the target fusion data, so that the facial expression and lip shape of the target virtual object correspond to the target object.

[0096] The technical solution of the disclosed embodiment can collect audio data and target part data of the target object in real time, determine lip shape data by analyzing and processing the audio data, and determine facial expression data by analyzing and processing the target part data, and further fuse the lip shape data and facial expression data to obtain target fusion data for driving the target virtual object. This solves the problem in the prior art that when driving the virtual object by expression, it is impossible to achieve real-time performance and thus there is a long lag, and realizes real-time analysis and processing of the collected data, thereby achieving the technical effect of synchronously driving the target virtual object.

[0097] The image processing device provided in the embodiments of the present disclosure can execute the image processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0098] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0099] Figure 4 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 4 , which shows an electronic device (eg, Figure 4 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0100] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An edit / output (I / O) interface 405 is also connected to the bus 404.

[0101] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0102] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0103] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0104] The electronic device provided by the embodiment of the present disclosure and the image processing method provided by the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0105] The embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method provided by the above embodiments is implemented.

[0106] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0107] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperTextTransferProtocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an adhoc peer-to-peer network), as well as any currently known or future developed network.

[0108] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0109] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0110] Acquire audio data and target part data corresponding to the target object; wherein the target part data corresponds to the position information and state information of the preset acquisition part;

[0111] Determining first data to be fused corresponding to the audio data; and,

[0112] Determining second data to be fused corresponding to the target part data;

[0113] Based on the first data to be fused and the second data to be fused, target fusion data is determined, so as to drive a target virtual object to be displayed based on the target fusion data.

[0114] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0115] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] According to one or more embodiments of the present disclosure, [Example 1] provides an image processing method, the method comprising:

[0117] Acquire audio data and target part data corresponding to the target object; wherein the target part data corresponds to the position information and state information of the preset acquisition part;

[0118] Determining first data to be fused corresponding to the audio data; and,

[0119] Determining second data to be fused corresponding to the target part data;

[0120] Based on the first data to be fused and the second data to be fused, target fusion data is determined, so as to drive a target virtual object to be displayed based on the target fusion data.

[0121] According to one or more embodiments of the present disclosure, [Example 2] provides an image processing method, the method comprising:

[0122] Optionally, the acquiring audio data and target part data corresponding to the target object includes:

[0123] In response to the virtual object driving operation, collecting audio information corresponding to the target object based on an audio collection device; and

[0124] Collecting target part data corresponding to the target object based on a facial image acquisition device;

[0125] The part corresponding to the target part data includes at least one part of the five sense organs.

[0126] According to one or more embodiments of the present disclosure, [Example 3] provides an image processing method, the method comprising:

[0127] Optionally, the determining the first data to be fused corresponding to the audio data includes:

[0128] Determine a plurality of text information corresponding to the audio data, and determine pronunciation information corresponding to the plurality of text information;

[0129] Based on the pronunciation information, first data to be fused of the mouth part is determined.

[0130] According to one or more embodiments of the present disclosure, [Example 4] provides an image processing method, the method comprising:

[0131] Optionally, the determining the second data to be fused corresponding to the target part includes:

[0132] Based on the target part data, determining grid point data of a plurality of grids corresponding to the target part;

[0133] The grid point data is used as the second data to be fused.

[0134] According to one or more embodiments of the present disclosure, [Example 5] provides an image processing method, the method further comprising:

[0135] Optionally, determining target fused data based on the first data to be fused and the second data to be fused includes:

[0136] Determine the maximum value of the first data to be fused and the second data to be fused corresponding to the same grid point, and use the maximum value as the target grid point data of the corresponding grid point;

[0137] The target fusion data is determined based on the target grid point data of at least part of the grid points.

[0138] According to one or more embodiments of the present disclosure, [Example 6] provides an image processing method, the method further comprising:

[0139] Optionally, determining the target fusion data based on target grid point data of at least part of the grid points includes:

[0140] Process the target grid point data according to the preset fusion curve to determine the fusion data to be superimposed corresponding to the target grid point;

[0141] The target fused data is updated based on the superimposed data to be fused.

[0142] According to one or more embodiments of the present disclosure, [Example 7] provides an image processing method, the method further comprising:

[0143] Optionally, the fusion curve corresponds to a preset expression.

[0144] According to one or more embodiments of the present disclosure, [Example 8] provides an image processing method, the method further comprising:

[0145] Optionally, the facial expression and lip shape of the target virtual object are updated based on the target fusion data, so that the facial expression and lip shape of the virtual object correspond to those of the target object.

[0146] According to one or more embodiments of the present disclosure, [Example 9] provides an image processing device, the device comprising:

[0147] A data acquisition module, used to acquire audio data and target part data corresponding to the target object; wherein the target part data corresponds to the position information and state information of the preset acquisition part;

[0148] A first data determination module is used to determine first data to be fused corresponding to the audio data; and

[0149] A second data determination module, used to determine second data to be fused corresponding to the target part data;

[0150] The target fusion data determination module is used to determine target fusion data based on the first data to be fused and the second data to be fused, so as to drive the target virtual object to be displayed based on the target fusion data.

[0151] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0152] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0153] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: Acquire audio data and target part data corresponding to the target object; wherein the target part data corresponds to the position information and state information of the preset acquisition part; Determining first data to be fused corresponding to the audio data; and, Determining second data to be fused corresponding to the target part data; Based on the first data to be fused and the second data to be fused, target fusion data is determined, so as to drive a target virtual object to be displayed based on the target fusion data.

2. The method according to claim 1, characterized in that The step of obtaining audio data and target part data corresponding to the target object includes: In response to the virtual object driving operation, collecting audio information corresponding to the target object based on an audio collection device; and Collecting target part data corresponding to the target object based on a facial image acquisition device; The part corresponding to the target part data includes at least one part of the five sense organs.

3. The method according to claim 1, characterized in that The determining the first data to be fused corresponding to the audio data includes: Determine a plurality of text information corresponding to the audio data, and determine pronunciation information corresponding to the plurality of text information; Based on the pronunciation information, first data to be fused of the mouth part is determined.

4. The method according to claim 1, characterized in that: The determining of the second data to be fused corresponding to the target part includes: Based on the target part data, determining grid point data of a plurality of grids corresponding to the target part; The grid point data is used as the second data to be fused.

5. The method according to claim 1, characterized in that The determining target fusion data based on the first data to be fused and the second data to be fused includes: Determine the maximum value of the first data to be fused and the second data to be fused corresponding to the same grid point, and use the maximum value as the target grid point data of the corresponding grid point; The target fusion data is determined based on the target grid point data of at least part of the grid points.

6. The method according to claim 5, characterized in that The determining the target fusion data based on the target grid point data of at least part of the grid points comprises: The target grid point data is adjusted according to a preset fusion curve to determine the target grid point fusion data to be superimposed; The target fusion data is updated based on the fusion data to be superimposed.

7. The method according to claim 6, characterized in that The fusion curve corresponds to a preset expression.

8. The method according to claim 1, characterized in that Also includes: The facial expression and the lip shape of the target virtual object are updated based on the target fusion data, so that the facial expression and the lip shape of the target virtual object correspond to the target object.

9. An image processing device, characterized in that: include: A data acquisition module, used to acquire audio data and target part data corresponding to the target object; A first data determination module, used to determine first data to be fused corresponding to the audio data; as well as, A second data determination module, used to determine second data to be fused corresponding to the target part data; The target fusion data determination module is used to determine target fusion data based on the first data to be fused and the second data to be fused, so as to drive the target virtual object to be displayed based on the target fusion data.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1 to 8.

11. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the image processing method according to any one of claims 1 to 8 when executed by a computer processor.

Images