Method, electronic device and computer program product for processing data

Abstract

Embodiments of the present disclosure relate to a method, an electronic device and a computer program product for processing data. The method comprises determining encoded data for a target object by hybrid encoding a plurality of types of data for the target object, the plurality of types of data including image data of the target object. The method further comprises determining an invariant part of a shape for the target object and a variable part of an expression for the target object by transforming the encoded data. The method improves data processing efficiency, saves time and computing resources, and reduces data transmission volume.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of data processing, and more specifically, to methods, electronic devices, and computer program products for processing data. Background Technology

[0002] With the development of computer technology, people have begun to utilize computer vision technology to obtain information about target objects or environments. This process is generally achieved by processing acquired images or videos. Computer vision processing includes image processing, pattern recognition or image recognition, scene analysis, and image understanding. Among these, image processing and object understanding are one of the core problems of computer vision. However, many problems still need to be solved in the process of visual image processing. Summary of the Invention

[0003] Embodiments of this disclosure provide a method, electronic device, and computer program product for processing data.

[0004] According to a first aspect of this disclosure, a method for processing data is provided. The method includes determining encoded data for a target object by performing mixed encoding on multiple types of data, including image data of the target object. The method further includes determining invariant portions of the shape of the target object and variable portions of the target object's facial expression by transforming the encoded data.

[0005] According to a second aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processor; and a memory coupled to the at least one processor and having instructions stored thereon, the instructions causing the device to perform actions when executed by the at least one processor, the actions including: determining encoded data for a target object by mixed encoding of multiple types of data for the target object, the multiple types of data including image data of the target object; and determining invariant portions of the shape of the target object and variable portions of the expression of the target object by transforming the encoded data.

[0006] According to a third aspect of this disclosure, a computer program product is provided, which is tangibly stored on a non-volatile computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform the steps of the method in the first aspect of this disclosure. Attached Figure Description

[0007] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0008] Figure 1 The illustration shows a schematic diagram of an example environment 100 in which the devices and / or methods according to embodiments of the present disclosure may be implemented;

[0009] Figure 2 A flowchart of a method 200 for processing data according to an embodiment of the present disclosure is illustrated;

[0010] Figure 3 The illustration shows a schematic diagram of an example process 300 for processing data using a cross-modal encoder according to an embodiment of the present disclosure;

[0011] Figure 4 The illustration shows a schematic diagram of an example process 400 for processing data according to an embodiment of the present disclosure;

[0012] Figure 5 The illustration shows a schematic diagram of an example 500 of a cross-modal architecture according to an embodiment of the present disclosure;

[0013] Figure 6 The illustration shows a schematic diagram of a process 600 for adjusting a cross-modal encoder according to an embodiment of the present disclosure;

[0014] Figure 7 The illustration shows a schematic diagram of the component configuration structure 700 according to an embodiment of the present disclosure;

[0015] Figure 8 A schematic block diagram of an example device 800 suitable for implementing embodiments of the present disclosure is shown.

[0016] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] With the development of technology, technicians have created many avatars. These avatars can be animated characters created by technicians, or they can be images of people obtained by users. To enhance the expressions of these avatars, computational technicians often apply the expressions of other objects to them.

[0020] In traditional solutions, video images of the target object are typically uploaded to an edge server or cloud server over a network. The edge server or cloud server then processes the uploaded video, identifying the corresponding facial expressions and combining them with the avatar. However, uploading the entire video image to the edge server or cloud server requires significant communication resources. Furthermore, processing the entire video on the edge server or cloud server consumes substantial computing resources and time.

[0021] To address at least the aforementioned and other potential problems, embodiments of this disclosure propose a method for processing data. In this method, a computing device determines encoded data for a target object by performing mixed encoding on multiple types of data, including image data of the target object. The computing device then determines the invariant portions of the target object's shape and the variable portions of its facial expressions by transforming the encoded data. This method improves data processing efficiency, saves time and computing resources, and reduces the amount of data transmitted.

[0022] The embodiments of this disclosure will now be described in further detail with reference to the accompanying drawings, wherein... Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown.

[0023] like Figure 1 As shown, the example environment 100 includes a computing device 106. The computing device 106 is used to receive various types of data 102 for processing.

[0024] The computing device 106 includes, but is not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multiprocessor systems, consumer electronics, minicomputers, mainframe computers, and distributed computing environments that include any of the above systems or devices.

[0025] The various types of data 102 in example environment 100 include image data 104 for a target object. In one example, the image data 104 is a video frame or an image. In another example, the image data 104 could be video. The above examples are merely for describing this disclosure and are not intended to specifically limit this disclosure.

[0026] The image data 104 is targeted at a specific object, such as a face, or a person or other suitable object. The various types of data 102 may also include at least one of the following: audio data and text data. Alternatively or additionally, the image data, audio data, and / or text data are mutually corresponding. For example, the image data may be a video frame or the video segment itself, while the audio and text data may be the audio and text data corresponding to that video segment.

[0027] The various types of data 102 are then input into the hybrid encoder 108 in the computing device 106 to obtain encoded data. In some embodiments, the hybrid encoder 108 includes a two-level structure. The first level structure consists of multiple encoders for encoding different types of data separately. The second level is a cross-modal encoder for uniformly encoding the multiple types of data encoded by the first level structure. In one example, the first level structure consists of three encoders for encoding text, audio, and image data, respectively. The second level structure is a cross-modal encoder for encoding the three types of encoded data. In some embodiments, the hybrid encoder 108 can be any suitable encoder capable of encoding multiple types of data. The above examples are merely for describing this disclosure and are not intended to specifically limit this disclosure.

[0028] The hybrid encoder 108 then inputs the encoded data into the transformation module 110. The transformation module performs matrix transformation on the encoded data to obtain the invariant part 112 and the variable part 114 for the target object.

[0029] In one example, if the input image is a human face with an expression, the invariant part corresponds to the shape of the face, such as the face shape and the distance between facial features, while the variable part corresponds to the expression, such as the size of the mouth opening and the size of the eyes opening. The above example is merely for describing this disclosure and not for limiting it specifically. The variable part 114 can then be applied to other objects or avatars. For example, the expression can be combined with other avatars in the avatar pool to generate an avatar with a new expression.

[0030] This method improves data processing efficiency, saves time and computing resources, and reduces data transmission volume.

[0031] The above combination Figure 1 A block diagram of an example system 100 in which embodiments of the present disclosure can be implemented is described. The following is in conjunction with… Figure 2 A flowchart describing a method 200 for processing data according to embodiments of the present disclosure is provided. Method 200 can be performed in... Figure 1 It is executed on computing device 106 and any suitable computing device.

[0032] At box 202, computing device 106 determines encoded data for a target object by performing mixed encoding on multiple types of data for the target object, including image data of the target object.

[0033] In some embodiments, the multiple types of data also include at least one of the following: voice data or text data. By using this method, more information can be obtained to determine a user's facial expressions using text or voice, thus improving the accuracy of the facial expression information.

[0034] In some embodiments, computing device 106 encodes each of a variety of data types. For example, text, audio, and video are encoded using a text encoder, an audio encoder, and a video encoder. The computing device 106 then performs a mixed encoding of the encoded data of the various types to determine encoded data. In one example, the encoded data is represented in matrix form. The above examples are merely illustrative of the present disclosure and not intended to limit its specific scope.

[0035] In some embodiments, the computing device 106 applies multiple types of data to a cross-modal encoder to perform mixed encoding of the multiple types of data. The cross-modal encoder is trained by using multiple types of sample data as input and labeled sample results as output. The training processes for the encoder for specific types of data and the cross-modal encoder are combined. Figure 5 and Figure 6 Describe it.

[0036] At box 204, computing device 106 determines the invariant parts of the shape of the target object and the variable parts of the expression of the target object by transforming the encoded data.

[0037] In some embodiments, the computing device 106 performs a first matrix transformation on the encoded data to determine an invariant portion. The computing device 106 also performs a second matrix transformation on the encoded data to determine a variable portion, the second matrix transformation being different from the first matrix transformation.

[0038] For example, for a matrix output by a hybrid encoder Where N represents the number of data points processed by the encoder, such as the number of points in a point cloud or the number of pixels in an image, and d is the dimension of each data point. Therefore, matrix factorization can be used to... Decomposed into matrices u and v, where u represents the invariant part and v represents the variable part, as shown below.

[0039]

[0040] Where u∈R N×k ,v∈R N×k, where k is a hidden dimension set by the user.

[0041] In order to pass After obtaining u and v, we use the Singular Value Decomposition (SVD) method to perform the decomposition. It can be expressed as the following formula:

[0042]

[0043] Therefore, u is eigenvalues, where v is The eigenvalues ​​of S are A single value, which is or The square root of . Therefore, the invariant part u and the variable part v can be obtained through the matrix transformation above.

[0044] Figure 3 An example of data processing is shown. For example... Figure 3 As shown, the aircraft image is processed by the cross-modal encoder 302, which outputs a vector matrix 304. Then, it is divided into an invariant part u 306 and a variable part v 308 through matrix decomposition. For the aircraft image, the invariant part is, for example, the coordinate system, and the variable part is the coordinate points of the aircraft.

[0045] Return now Figure 2 The following description, in some embodiments, shows that the variable portion can be applied to other objects. (The following is in conjunction with...) Figure 4 A schematic diagram illustrating an example process 400 representing an application variation according to embodiments of the present disclosure. Process 400 can be... Figure 1 It is executed on computing device 106 and any suitable computing device.

[0046] First, the video 402, audio 404, and text 406 are processed by their respective encoders and then input into the cross-modal encoder 408. The cross-modal encoder 408 then generates the variable representation 410, i.e., the variable part, and then combines the variable representation into the object or avatar 412.

[0047] This method improves data processing efficiency, saves time and computing resources, and reduces data transmission volume.

[0048] The above combination Figure 2-4 A flowchart of a method 200 for processing data according to an embodiment of the present disclosure is described below. (See section 5 for further details.) Figure 6 The training process of encoders and cross-modal encoders for processing various types of data is described in embodiments of this disclosure. Figure 5A schematic diagram of Example 500 for training an encoder according to embodiments of the present disclosure is described.

[0049] The cross-modal architecture includes a text encoder 508, an audio encoder 510, a video encoder 512, a cross-modal encoder 514, a text decoder 516, an audio decoder 518, and a video decoder 520. During training, text data 502, audio data 504, and video data 506, used as sample inputs, are input to the text encoder 508, audio encoder 510, and video encoder 512 for encoding. The encoded data is then input to the cross-modal encoder 514 for further encoding, and then to the text decoder 516, audio decoder 518, and video decoder 520 for decoding. The obtained data is compared with the input data, and the encoder parameters are adjusted until the input and output are consistent.

[0050] The training process employs three levels of input adjustment. The first level is token-level masking, which masks a specific data element in the input data before using the processed data as input for training. The second level is modality-level adjustment, which masks the input of one of the encoders in the text decoder 516, audio decoder 518, and video decoder 520 for training. The third level is sampling-level adjustment, which matches or dismatches the three types of data input to the cross-modal encoder 514 for training. After this training, a text encoder, audio encoder, video encoder, and cross-modal encoder usable in this disclosure can be obtained. Figure 5 This is just an example; a cross-modal encoder for any type of data can be achieved through... Figure 5 It was obtained through training.

[0051] Figure 6 A schematic diagram illustrating an example 600 for adjusting a cross-modal encoder according to an embodiment of the present disclosure.

[0052] The cross-modal encoder 602 is in Figure 5 The image is pre-trained. By processing the aircraft image using the cross-modal encoder 602, matrix 606 can be obtained. Then, matrix decomposition is performed to obtain u1 610 and v1 616, which represent invariant information and variable information, respectively.

[0053] The rotated aircraft image is then input into an enhanced transmodal encoder 604, which is initially the same as the transmodal encoder 602, or the initial enhanced transmodal encoder 604 is the transmodal encoder 602.

[0054] After processing the rotated image through the enhanced cross-modal encoder 604, a second matrix 608 is obtained. This matrix is ​​then decomposed to obtain the invariant part u2614 and the variable median v2620. The similarity loss 612 between u1610 and u2614 can then be calculated.

[0055] L S =sim(u1, u2)

[0056] Where sim() represents the similarity metric, for example, it can be cosine similarity.

[0057] Then, for v1 616 and v2 620, the similarity loss 618 of the transformation is calculated as follows:

[0058] L v = -sim(v1, trans(v2))

[0059] Here, `trans()` represents a linear transformation function, which can also be learned and adjusted. Then, the overall loss is determined based on the two losses mentioned above:

[0060] L = L s +L v

[0061] By adjusting the coefficients of trans() and the parameters of the enhanced cross-modal encoder 604, the value of the overall loss function is made less than a predetermined threshold to obtain a suitable trans() function and an enhanced cross-modal encoder.

[0062] This enhanced cross-modal encoder, combined with previously trained encoders for different types of data, forms a pre-trained encoder. In application, the pre-trained encoder is used to process the input data, followed by matrix transformation to obtain the changed components.

[0063] This method improves data processing efficiency, saves time and computing resources, and reduces data transmission volume.

[0064] The above combines 5 and Figure 6 The training process of encoders and cross-modal encoders for processing various types of data is described in embodiments of this disclosure. The following, in conjunction with... Figure 7 A schematic diagram illustrating the component configuration structure 700 according to an embodiment of the present disclosure.

[0065] In Figure 700, terminal device 702 acquires audio 708, video 710, and files 712, then transmits this data to cloud data center 704. The data is stored in storage device 714 and then processed by dedicated computing resources 716 in the cloud data center, such as a GPU, to obtain the variable portion 718 and the invariant portion 720 of the data. Only the variable portion 718 is transmitted to edge device 706, where it is combined with avatars in avatar pool 724 via avatar rendering module 722 and provided to edge programming interface (API) connector 726. The API connector 726 then provides the data to the corresponding application.

[0066] This arrangement reduces the amount of data transmitted, speeds up data processing, and improves efficiency.

[0067] Figure 8 A schematic block diagram of an example device 800 that can be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device 106 can be implemented using device 800. As shown, device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 can also store various programs and data required for the operation of device 800. CPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0068] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage page 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0069] The various processes and procedures described above, such as methods 200, 300, 400, and 600, may be executed by processing unit 801. For example, in some embodiments, methods 200, 300, 400, and 600 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions of methods 200, 300, 400, and 600 described above may be performed.

[0070] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0071] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0072] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0073] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0074] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0075] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0076] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0077] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0078] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing data, comprising: Encoded data for the target object is determined by mixing and encoding multiple types of data for the target object, including image data of the target object. as well as By transforming the encoded data, the invariant parts of the shape of the target object and the variable parts of the expression of the target object are determined; Determining the invariant portion and the variable portion includes: Perform a first matrix transformation on the encoded data to determine the invariant portion; as well as A second matrix transformation is performed on the encoded data to determine the variable portion, the second matrix transformation being different from the first matrix transformation.

2. The method of claim 1, wherein determining the encoded data comprises: Encode each of the multiple data types; as well as Multiple types of encoded data are mixed-encoded to determine the encoded data.

3. The method according to claim 1, wherein determining the encoded data comprises: Multiple types of data are applied to a cross-modal encoder to encode the multiple types of data in a mixed manner. The cross-modal encoder is trained by using multiple types of sample data as input and labeled sample results as output.

4. The method according to claim 1, wherein the multiple types of data further include at least one of the following: voice data or text data.

5. The method of claim 1, wherein the variable portion may be applied to other objects.

6. An electronic device, comprising: At least one processor; as well as A memory, coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, cause the electronic device to perform actions, including: Encoded data for the target object is determined by performing mixed encoding on multiple types of data targeting the target object, including image data of the target object; and By transforming the encoded data, the invariant parts of the shape of the target object and the variable parts of the expression of the target object are determined; Determining the invariant portion and the variable portion includes: Perform a first matrix transformation on the encoded data to determine the invariant portion; and A second matrix transformation is performed on the encoded data to determine the variable portion, the second matrix transformation being different from the first matrix transformation.

7. The electronic device of claim 6, wherein determining the encoded data comprises: Encode each of the multiple data types; as well as Multiple types of encoded data are mixed-encoded to determine the encoded data.

8. The electronic device of claim 6, wherein determining the encoded data comprises: Multiple types of data are applied to a cross-modal encoder to encode the multiple types of data in a mixed manner. The cross-modal encoder is trained by using multiple types of sample data as input and labeled sample results as output.

9. The electronic device of claim 6, wherein the plurality of data further includes at least one of the following: voice data or text data.

10. The electronic device of claim 6, wherein the variable portion may be applied to other objects.

11. A computer program product tangibly stored on a non-volatile computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the steps of the method according to any one of claims 1 to 5.

Images