Methods, apparatus, devices and storage media for generating dynamic human images

Abstract

This application discloses a method, apparatus, device, and storage medium for generating dynamic human images, belonging to the field of visual recognition technology. The method includes: processing video images of the face and body parts of a person engaged in conversation within a vehicle using a target face recognition model and a target body recognition model to obtain facial feature points and body part feature points; determining the position coordinates of facial reference points and body part reference points in a two-dimensional animation vector image based on the position coordinates of the facial and body part feature points; determining the facial contour curve and body contour curve in the two-dimensional animation vector image based on the position coordinates of the facial and body part reference points; and determining the two-dimensional animation of the person engaged in conversation based on the facial and body contour curves in the two-dimensional animation vector image. Two-dimensional animation has a smaller data volume than video, reducing the requirements for memory, transmission capacity, and transmission costs of in-vehicle equipment.

Description

Technical Field

[0001] This application relates to the field of visual recognition technology, and in particular to a method, apparatus, device and storage medium for generating dynamic human images. Background Technology

[0002] With the development of vehicle intelligence, visual recognition technology is being applied to various vehicle functions. When passengers are talking inside the vehicle, the facial expressions and movements of the people talking are dynamically displayed, which is very beneficial to improving the interactivity of the conversation.

[0003] In related technologies, real-time video is obtained by recording the facial expressions and actions of the people talking, and then the real-time video is transmitted to the vehicle system for display.

[0004] In related technologies, due to the large amount of data in real-time video, the requirements for the memory, transmission capacity, and transmission cost of the vehicle-mounted equipment are high during the transmission of real-time video. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for generating dynamic human images, which can be used to solve problems existing in related technologies. The technical solution is as follows:

[0006] On one hand, embodiments of this application provide a method for generating dynamic human images, the method comprising:

[0007] The system acquires video images of people conversing inside the vehicle, including video images of the faces and body parts of the people conversing.

[0008] The video image of the face is processed by the target face recognition model to obtain a first number of facial feature points, and the video image of the body part is processed by the target body recognition model to obtain a second number of body part feature points.

[0009] Based on the position coordinates of the first number of facial feature points, determine the position coordinates of the first number of facial reference points in the two-dimensional animation vector image; based on the position coordinates of the second number of body part feature points, determine the position coordinates of the second number of body part reference points in the two-dimensional animation vector image.

[0010] Based on the position coordinates of the first number of facial reference points, the facial contour curve in the two-dimensional animation vector image is determined; based on the position coordinates of the second number of body part reference points, the body part contour curve in the two-dimensional animation vector image is determined.

[0011] The 2D animation of the dialogue character is determined based on the facial contour curve and the body part contour curve in the 2D animation vector image.

[0012] On the other hand, a device for generating dynamic human images is provided, the device comprising:

[0013] The acquisition module is used to acquire video images of people conversing inside the vehicle. The video images include video images of the faces of the people conversing and video images of the body parts of the people conversing.

[0014] The recognition module is used to perform facial recognition processing on the video image of the face through the target face recognition model to obtain a first number of facial feature points, and to perform body part recognition processing on the video image of the body part through the target body recognition model to obtain a second number of body part feature points.

[0015] The first determining module is used to determine the position coordinates of a first number of facial reference points in the two-dimensional animation vector image based on the position coordinates of the first number of facial feature points, and to determine the position coordinates of a second number of body part reference points in the two-dimensional animation vector image based on the position coordinates of the second number of body part feature points.

[0016] The second determining module is used to determine the facial contour curve in the two-dimensional animation vector image based on the position coordinates of the first number of facial reference points, and to determine the body part contour curve in the two-dimensional animation vector image based on the position coordinates of the second number of body part reference points.

[0017] The third determining module is used to determine the two-dimensional animation of the dialogue character based on the facial contour curve and the body part contour curve in the two-dimensional animation vector image.

[0018] On the other hand, a dynamic portrait generation device is provided, the dynamic portrait generation device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor, so that the dynamic portrait generation device implements any of the dynamic portrait generation methods described above.

[0019] On the other hand, a computer-readable storage medium is also provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to enable a computer to implement any of the above-described methods for generating dynamic human figures.

[0020] On the other hand, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of the dynamic portrait generation device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the dynamic portrait generation device to perform any of the dynamic portrait generation methods described above.

[0021] The technical solution provided in this application has at least the following beneficial effects:

[0022] This application acquires video images of the faces and body parts of characters engaged in dialogue inside a vehicle. These images are then processed using a target face recognition model and a target body recognition model to obtain facial and body feature points. A 2D animation vector diagram is then created. Based on the positional coordinates of the facial and body feature points, the positional coordinates of facial and body reference points within the 2D animation vector diagram are determined. Based on these coordinates, the facial and body contour curves within the 2D animation vector diagram are then determined, thereby defining the 2D animation of the characters in dialogue. Because 2D animation has a smaller data volume than video, it reduces the requirements for the memory, transmission capacity, and transmission costs of in-vehicle equipment. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0025] Figure 2 This is a flowchart of a method for generating a dynamic human image provided in an embodiment of this application;

[0026] Figure 3 This is a distribution map of facial feature points provided in an embodiment of this application;

[0027] Figure 4 This is a distribution map of feature points of a body part provided in an embodiment of this application;

[0028] Figure 5 This is a function graph of a first Bézier curve provided in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram illustrating the generation process of a dynamic portrait of a person in a remote dialogue, as provided in an embodiment of this application.

[0030] Figure 7 This is a schematic diagram illustrating the generation process of a dynamic portrait of a person in direct dialogue, as provided in an embodiment of this application.

[0031] Figure 8 This is a schematic diagram of the structure of a dynamic human image generation device provided in an embodiment of this application;

[0032] Figure 9 This is a schematic diagram of the structure of a server provided in an embodiment of this application;

[0033] Figure 10 This is a schematic diagram of the structure of a dynamic human image generation device provided in an embodiment of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0035] This application provides a method for generating dynamic human portraits. Please refer to... Figure 1 The diagram illustrates an implementation environment for the method provided in this embodiment. This implementation environment may include a vehicle 11 and a terminal 12.

[0036] Specifically, terminal 12 acquires video images of the faces and body parts of the characters engaged in conversation inside vehicle 11. These images are then processed using a target face recognition model and a target body recognition model on mobile phone 11 to obtain a first set of facial feature points and a second set of body part feature points. Terminal 12 determines a 2D animation vector image. Based on the position coordinates of the facial feature points, it determines the position coordinates of facial reference points in the 2D animation vector image. Similarly, based on the position coordinates of the body part feature points, it determines the position coordinates of body part reference points in the 2D animation vector image. After determining the position coordinates of the facial and body part reference points, terminal 12 can determine the facial and body contour curves in the 2D animation vector image, thereby determining the 2D animation of the characters engaged in conversation inside vehicle 11. Terminal 12 can store the 2D animation of the characters engaged in conversation, and vehicle 11 can retrieve and display the 2D animation from terminal 12. Alternatively, vehicle 11 can also store the 2D animation of the characters engaged in conversation.

[0037] Optionally, the terminal 12 can be a mobile phone or a computer, and the vehicle 11 establishes a communication connection with the terminal 12 via a wireless network or Bluetooth.

[0038] Optionally, the implementation environment may also include server 13, which may be a server. Vehicle 11 and terminal 12 establish communication connections with server 13 via a wireless network.

[0039] Based on the above Figure 1 The implementation environment shown in this application embodiment provides a method for generating dynamic human images, such as... Figure 2 As shown, the method for generating the dynamic human image can be executed by the terminal, and the method includes steps 201-205.

[0040] In step 201, video images of the people conversing inside the vehicle are acquired. The video images include video images of the faces of the people conversing and video images of the body parts of the people conversing.

[0041] This application does not limit the device used to capture video images of people conversing inside a vehicle. For example, a mobile phone can be used to capture video images of people conversing inside a vehicle.

[0042] In one possible implementation, the video images include video images of the faces of the people in the conversation and video images of their body parts. Acquiring video images of the people in the conversation inside the vehicle includes: acquiring video images of the faces and body parts of the people in the conversation inside the vehicle using a mobile phone inside the vehicle. The mobile phone inside the vehicle can be a phone held by one of the people in the conversation or a phone held by someone other than the people in the conversation inside the vehicle.

[0043] It can also capture video images of people conversing outside the vehicle, who can then communicate with people inside the vehicle via a remote communication tool. This application does not limit the remote communication tool; for example, it can be a mobile phone belonging to someone outside the vehicle.

[0044] In one possible implementation, acquiring video images of the people in the conversation further includes: acquiring video images of the faces and body parts of the people in the conversation outside the vehicle using a mobile phone located outside the vehicle. The mobile phone outside the vehicle can be a phone held by the person in the conversation or a phone held by someone other than the person in the conversation outside the vehicle.

[0045] The video formats captured by the mobile phone include, but are not limited to, MP4 (Moving Pictures Experts Group-4, portable media player) format.

[0046] In step 202, the video image of the face is processed by the target face recognition model to obtain a first number of facial feature points, and the video image of the body part is processed by the target body recognition model to obtain a second number of body part feature points.

[0047] Optionally, before performing facial recognition processing on the video image of the face using the target facial recognition model, a target facial recognition model is first established. Methods for establishing the target facial recognition model include, but are not limited to: acquiring a facial training image and a corresponding facial reference image, the facial reference image including a first number of labeled facial feature points; determining the first number of facial feature points on the facial training image based on the initial facial recognition model; determining a loss value based on the labeled facial feature points on the facial reference image and the facial feature points on the facial training image; and adjusting the initial facial recognition model based on the loss value to obtain the target facial recognition model.

[0048] This application does not limit the first quantity in its embodiments. For example, the first quantity can be set based on experience or adjusted according to actual conditions. This application also does not limit the initial face recognition model. The initial face recognition model can be constructed based on the annotation of facial feature points in the facial reference image, or it can be adjusted according to actual conditions.

[0049] For example, after constructing the initial face recognition model, the face training image is imported into the initial face recognition model. After the initial face recognition model performs face recognition processing on the face training image, the first number of facial feature points in the face training image are obtained. The positions of the facial feature points in the face training image are compared with the positions of the facial feature points in the face reference image to determine the loss value generated by the initial face recognition model. Based on the loss value, the initial face recognition model is adjusted so that the positions of the facial feature points in the face training image obtained after the next face recognition model processing are more likely to coincide with the positions of the facial feature points in the face reference image.

[0050] This application does not limit the method of adjusting the initial face recognition model based on the loss value. For example, the number of iterations can be set based on experience, and the face recognition model obtained when the number of iterations reaches the set number can be used as the target face recognition model. Alternatively, the face recognition model can be adjusted based on the loss value. A reference loss value can be set based on experience. When the loss value between the positions of facial feature points in the training image and the positions of facial feature points in the reference image is less than the reference loss value, the face recognition model obtained at this time can be used as the target face recognition model.

[0051] In one possible implementation, after constructing the target face recognition model, the video image of the face is processed by the target face recognition model to obtain a first number of facial feature points. This includes: importing the acquired video image of the speaker's face into the target face recognition model; after the target face recognition model processes the face, a first number of facial feature points are obtained from the video image of the speaker's face; the distribution of the facial feature points is as follows: Figure 3 As shown.

[0052] Similarly, before performing body part recognition processing on video images of body parts using the target body recognition model, a target body recognition model is first established. Methods for establishing the target body recognition model include, but are not limited to: acquiring training images of body parts and corresponding reference images of body parts, the reference images including a second set of labeled body part feature points; determining the second set of labeled body part feature points on the training images based on the initial body recognition model; determining a loss value based on the labeled body part feature points on the reference images and the body part feature points on the training images; and adjusting the initial body recognition model based on the loss value to obtain the target body recognition model.

[0053] This application embodiment does not impose any limitation on the second quantity. For example, the second quantity can be set based on experience or adjusted according to actual conditions. This application embodiment also does not impose any limitation on the initial body recognition model. The initial body recognition model can be constructed based on the annotation of body part feature points in the body part reference image, or it can be adjusted according to actual conditions.

[0054] For example, after constructing the initial body recognition model, the training images of body parts are imported into the initial body recognition model. After the initial body recognition model performs body part recognition processing on the training images of body parts, a second number of body part feature points are obtained in the training images of body parts. The positions of the body part feature points in the training images of body parts are compared with the positions of the body part feature points in the reference images of body parts, and the loss value generated by the initial body recognition model is determined. Based on the loss value, the initial body recognition model is adjusted so that the positions of the body part feature points in the training images of body parts obtained after the next processing by the body recognition model are more likely to coincide with the positions of the body part feature points in the reference images of body parts.

[0055] This application does not limit the number of times the body recognition model is adjusted. It can be set based on experience, or it can be adjusted according to the degree of overlap between the body part feature points in the training image and the body part feature points in the reference image. After adjustment, the target body recognition model is finally obtained.

[0056] In one possible implementation, after constructing the target body recognition model, the video images of body parts are processed by the target body recognition model to obtain a second number of body part feature points. This includes: importing the acquired video images of the dialogue participants' body parts into the target body recognition model; after the target body recognition model processes the body parts, the second number of body part feature points in the video images of the dialogue participants' body parts are obtained. The distribution of the body part feature points is as follows: Figure 4 As shown.

[0057] In step 203, the position coordinates of the first number of facial reference points in the two-dimensional animation vector image are determined based on the position coordinates of the first number of facial feature points, and the position coordinates of the second number of body part reference points in the two-dimensional animation vector image are determined based on the position coordinates of the second number of body part feature points.

[0058] In one possible implementation, before determining the position coordinates of the first number of facial reference points in the two-dimensional animation vector image based on the position coordinates of the first number of facial feature points, and before determining the position coordinates of the second number of body part reference points in the two-dimensional animation vector image based on the position coordinates of the second number of body part feature points, a two-dimensional animation vector image is first determined. This application embodiment does not limit the two-dimensional animation vector image. For example, the two-dimensional animation vector image can be a two-dimensional cartoon character image in SVG (Scalable Vector Graphics) format, and the two-dimensional cartoon image needs to include a face and body parts.

[0059] Any point in the video image of the faces of the people in the dialogue is determined as a reference point. A coordinate system is established with the reference point as the origin, and the horizontal axis of the coordinate system of the facial video image is the x-axis, and the vertical axis is the y-axis. This embodiment does not limit the choice of the reference point; it can be the center point of the face of the person in the dialogue. After determining the coordinate system, the position coordinates of the first number of facial feature points are determined based on their positions in the facial video image.

[0060] In one possible implementation, determining the position coordinates of a first number of facial reference points in a two-dimensional animation vector image based on the position coordinates of a first number of facial feature points includes: using the position coordinates of the first number of facial feature points as the position coordinates of the first number of facial reference points in the coordinate system of the video image of the face.

[0061] Similarly, any point in the video image of the body part of the person in the dialogue is determined as a reference point, and a coordinate system is established with the reference point as the origin. The horizontal axis of the coordinate system of the body part video image is the x-axis, and the vertical axis is the y-axis. This embodiment does not limit the choice of the reference point; it can be the center point of the body part of the person in the dialogue. After determining the coordinate system, the coordinates of the second number of body part feature points are determined based on their positions in the video image of the body part.

[0062] In one possible implementation, determining the position coordinates of a second number of body part reference points in a two-dimensional animation vector image based on the position coordinates of a second number of body part feature points includes: using the position coordinates of the second number of body part feature points as the position coordinates of the second number of body part reference points in the coordinate system of the video image of the body part.

[0063] In step 204, the facial contour curve in the two-dimensional animation vector image is determined based on the position coordinates of a first number of facial reference points, and the body part contour curve in the two-dimensional animation vector image is determined based on the position coordinates of a second number of body part reference points.

[0064] In one possible implementation, after determining the position coordinates of a first number of facial reference points, the facial contour curve in the 2D animation vector image is determined based on the position coordinates of the first number of facial reference points. This includes: substituting the position coordinates of the first number of facial reference points into a function of a first Bézier curve to obtain the control point coordinates of the first Bézier curve; determining the first Bézier curve based on the control point coordinates of the first Bézier curve, and using the first Bézier curve as the facial contour curve in the 2D animation vector image; wherein, the function of the first Bézier curve is:

[0065]

[0066] P1(t) is a function of the first Bézier curve, t is the independent variable of the function of the first Bézier curve, x(t) is the abscissa of the facial reference point, and y(t) is the ordinate of the facial reference point. i Let y be the x-coordinate of the control point of the first Bézier curve. i B is the ordinate of the control point of the first Bézier curve. i,n (t) is a Bernstein polynomial.

[0067] Where n is the order of the function of the first Bézier curve, which is equal to the number of control points controlling the first Bézier curve minus one, and i is a positive integer from 0 to n. Bernstein polynomial B... i,n (t) can be expanded using the following formula:

[0068]

[0069] For example, taking 4 control points as an example, when the control points are 4, n equals 3, and i is 0, 1, 2, or 3.

[0070] Therefore, when the control point is 4, the function graph of the first Bézier curve is as follows: Figure 5 As shown, P1(t) is as follows:

[0071]

[0072] By substituting the abscissa of the facial reference point into x(t) and the ordinate of the facial reference point into y(t), the coordinates of the control points of the first Bézier curve (x1, y1), (x2, y2), (x3, y3), and (x4, y4) can be calculated. Based on the coordinates of the control points of the first Bézier curve, the first Bézier curve P1(t) is determined. Among them, the two control points of the first Bézier curve with coordinates (x1, y1) and (x4, y4) are the two endpoints of the first Bézier curve. After determining the first Bézier curve, the first Bézier curve is used as the facial contour curve in the two-dimensional animation vector graphic.

[0073] Similarly, after determining the position coordinates of a second number of body part reference points, the outline curve of the body part in the 2D animation vector graphic is determined based on the position coordinates of the second number of body part reference points. This includes: substituting the position coordinates of the second number of body part reference points into the function of the second Bézier curve to obtain the control point coordinates of the second Bézier curve; determining the second Bézier curve based on the control point coordinates of the second Bézier curve, and using the second Bézier curve as the outline curve of the body part in the 2D animation vector graphic; wherein, the function of the second Bézier curve is:

[0074]

[0075] P2(t) is a function of the second Bézier curve, t is the independent variable of the function of the second Bézier curve, x(t) is the x-coordinate of the body part reference point, and y(t) is the y-coordinate of the face reference point. i Let y be the x-coordinate of the control point of the second Bézier curve. i B is the ordinate of the control point of the second Bézier curve. i,n (t) is a Bernstein polynomial.

[0076] Where n is the order of the function of the second Bézier curve, which is equal to the number of control points controlling the second Bézier curve minus one, and i is a positive integer from 0 to n. Bernstein polynomial B... i,n (t) can be expanded using the following formula:

[0077]

[0078] For example, taking 4 control points as an example, when the control points are 4, n equals 3, and i is 0, 1, 2, or 3.

[0079] Therefore, when the control point is 4, P2(t) is as follows:

[0080]

[0081] By substituting the abscissa of the body part reference point into x(t) and the ordinate of the facial reference point into y(t), the coordinates of the control points of the second Bézier curve (x1, y1), (x2, y2), (x3, y3), and (x4, y4) can be calculated. Based on the coordinates of the control points of the first Bézier curve, the second Bézier curve P2(t) is determined. The two control points of the second Bézier curve with coordinates (x1, y1) and (x4, y4) are the two endpoints of the second Bézier curve. After determining the second Bézier curve, it is used as the outline curve of the body part in the 2D animation vector graphic.

[0082] In step 205, the 2D animation of the dialogue character is determined based on the facial contour curve and the body part contour curve in the 2D animation vector image.

[0083] In one possible implementation, after determining the facial contour curve and the body part contour curve in the 2D animation vector image, the 2D animation of the dialogue character is determined based on the facial contour curve and the body part contour curve in the 2D animation vector image, including: combining the facial contour curve and the body part contour curve in the 2D animation vector image according to the 2D animation vector image to obtain the 2D animation of the dialogue character.

[0084] Optionally, after obtaining the 2D animation of the dialogue characters, the dialogue mode of the dialogue characters is obtained, including direct dialogue and remote dialogue; if the dialogue mode is direct dialogue, the 2D animation is transmitted to the vehicle's display screen via either the vehicle's internal wireless network or Bluetooth for display; if the dialogue mode is remote dialogue, the 2D animation is transmitted to the vehicle's display screen via the vehicle's internal Bluetooth for display.

[0085] Among them, obtaining the dialogue mode of the dialogue participants includes: if the dialogue participants include dialogue participants outside the vehicle, the dialogue mode includes remote dialogue and direct dialogue; if the dialogue participants do not include dialogue participants outside the vehicle, the dialogue mode includes direct dialogue.

[0086] In one possible implementation, when the dialogue method includes both remote and direct dialogue, the 2D animation of the characters in the dialogue inside the vehicle is transmitted to the vehicle's display screen via either the vehicle's Wi-Fi or Bluetooth, using the characters' mobile phones inside the vehicle. Conversely, the 2D animation of the characters outside the vehicle is transmitted to the mobile phones of the characters inside the vehicle, and then transmitted via Bluetooth from the mobile phones of the characters inside the vehicle to the vehicle's display screen for display.

[0087] When the dialogue method includes direct dialogue, the two-dimensional animation of the dialogue characters inside the vehicle is transmitted to the vehicle's display screen via either the vehicle's internal Wi-Fi or Bluetooth for display.

[0088] This application does not limit the vehicle's display screen. For example, taking the display screen of the vehicle's instrument display system as an example, a two-dimensional animation of a dialogue character can be displayed on the display screen of the vehicle's instrument display system.

[0089] In this embodiment, video images of the faces and body parts of the characters in a conversation inside a vehicle are acquired. These images are then processed using a target face recognition model and a target body recognition model to obtain facial and body feature points. A two-dimensional animation vector diagram is then created. Based on the position coordinates of the facial and body feature points, the position coordinates of facial and body reference points within the two-dimensional animation vector diagram are determined. Based on these position coordinates, the facial and body contour curves within the two-dimensional animation vector diagram are determined, thereby defining the two-dimensional animation of the characters in the conversation. Because two-dimensional animation has a smaller data volume than video, it reduces the requirements for the memory, transmission capacity, and transmission costs of the in-vehicle equipment.

[0090] Figure 6 This is a schematic diagram illustrating the generation process of a dynamic portrait of a person in a remote dialogue, as provided in an embodiment of this application. The process involves capturing video images of the face and body parts of the person in the dialogue outside the vehicle using a mobile phone 601. A target face recognition model 602 performs facial recognition processing on the facial video images to obtain a first number of facial feature points, and a target body recognition model 603 performs body part recognition processing on the body part video images to obtain a second number of body part feature points.

[0091] The position coordinates of facial reference points in the 2D animation vector graphic 604 are determined based on the position coordinates of facial feature points, and the position coordinates of body part reference points in the 2D animation vector graphic 604 are determined based on the position coordinates of body part feature points. Based on the position coordinates of the facial reference points, the facial contour curve 605 in the 2D animation vector graphic 604 is determined, and based on the position coordinates of the body part reference points, the body part contour curve 606 in the 2D animation vector graphic is determined. Based on the facial contour curve 605 and the body part contour curve 606, the 2D animation 607 of the dialogue character outside the vehicle is determined. The 2D animation 607 of the dialogue character outside the vehicle is transmitted to the mobile phone 608 of the dialogue character inside the vehicle, and then transmitted via Bluetooth 609 to the vehicle's display screen 610 for display.

[0092] Figure 7 This is a schematic diagram illustrating the generation process of a dynamic portrait of a person in direct conversation, as provided in an embodiment of this application. Specifically, video images of the face and body parts of the person in conversation are captured by a mobile phone 701 inside the vehicle. A target face recognition model 702 performs facial recognition processing on the facial video images to obtain a first number of facial feature points, and a target body recognition model 703 performs body part recognition processing on the body part video images to obtain a second number of body part feature points.

[0093] The position coordinates of facial reference points in the 2D animation vector graphic 704 are determined based on the position coordinates of facial feature points, and the position coordinates of body part reference points in the 2D animation vector graphic 704 are determined based on the position coordinates of body part feature points. Based on the position coordinates of the facial reference points, the facial contour curve 705 in the 2D animation vector graphic 704 is determined, and based on the position coordinates of the body part reference points, the body part contour curve 706 in the 2D animation vector graphic is determined. Based on the facial contour curve 705 and the body part contour curve 706, the 2D animation 707 of the characters in dialogue inside the vehicle is determined. The 2D animation 707 of the characters in dialogue inside the vehicle is transmitted to the vehicle's display screen 709 for display via the vehicle's internal wireless network 708.

[0094] See Figure 8 This application provides a device for generating dynamic human images, the device comprising:

[0095] The acquisition module 801 is used to acquire video images of people talking inside the vehicle. The video images include video images of the faces of the people talking and video images of the body parts of the people talking.

[0096] The recognition module 802 is used to perform facial recognition processing on the video image of the face through the target face recognition model to obtain a first number of facial feature points, and to perform body part recognition processing on the video image of the body part through the target body recognition model to obtain a second number of body part feature points.

[0097] The first determining module 803 is used to determine the position coordinates of a first number of facial reference points in a two-dimensional animation vector image based on the position coordinates of a first number of facial feature points, and to determine the position coordinates of a second number of body part reference points in a two-dimensional animation vector image based on the position coordinates of a second number of body part feature points.

[0098] The second determining module 804 is used to determine the facial contour curve in the two-dimensional animation vector image based on the position coordinates of the first number of facial reference points, and to determine the body part contour curve in the two-dimensional animation vector image based on the position coordinates of the second number of body part reference points.

[0099] The third determining module 805 is used to determine the two-dimensional animation of the dialogue character based on the facial contour curve and the body part contour curve in the two-dimensional animation vector image.

[0100] In one possible implementation, the device further includes:

[0101] The first acquisition module is used to acquire a facial training image and a facial reference image corresponding to the facial training image. The facial reference image includes a first number of labeled facial feature points.

[0102] The fourth determination module is used to determine a first number of facial feature points on the facial training image based on the initial face recognition model;

[0103] The first adjustment module is used to determine the loss value based on the facial feature points marked on the facial reference image and the facial feature points on the facial training image, and adjust the initial face recognition model based on the loss value to obtain the target face recognition model.

[0104] In one possible implementation, the device further includes:

[0105] The second acquisition module is used to acquire body part training images and body part reference images corresponding to the body part training images. The body part reference images include a second number of labeled body part feature points.

[0106] The fifth determining module is used to determine a second number of body part feature points on the body part training image based on the initial body recognition model;

[0107] The second adjustment module is used to determine the loss value based on the body part feature points marked on the body part reference image and the body part feature points on the body part training image, and adjust the initial body recognition model based on the loss value to obtain the target body recognition model.

[0108] In one possible implementation, the second determining module 804 is used to input the position coordinates of a first number of facial reference points into the function of the first Bézier curve to obtain the control point coordinates of the first Bézier curve; based on the control point coordinates of the first Bézier curve, the first Bézier curve is determined, and the first Bézier curve is used as the facial contour curve in the two-dimensional animation vector graphic; wherein, the function of the first Bézier curve is:

[0109]

[0110] P1(t) is a function of the first Bézier curve, t is the independent variable of the function of the first Bézier curve, x(t) is the abscissa of the facial reference point, and y(t) is the ordinate of the facial reference point. i Let y be the x-coordinate of the control point of the first Bézier curve. i B is the ordinate of the control point of the first Bézier curve. i,n (t) is a Bernstein polynomial.

[0111] In one possible implementation, the second determining module 804 is used to input the position coordinates of a second number of body part reference points into the function of the second Bézier curve to obtain the control point coordinates of the second Bézier curve; based on the control point coordinates of the second Bézier curve, the second Bézier curve is determined, and the second Bézier curve is used as the body part contour curve in the two-dimensional animation vector graphic; wherein, the function of the second Bézier curve is:

[0112]

[0113] P2(t) is a function of the second Bézier curve, t is the independent variable of the function of the second Bézier curve, x(t) is the x-coordinate of the body part reference point, and y(t) is the y-coordinate of the body part reference point. i Let y be the x-coordinate of the control point of the second Bézier curve. i B is the ordinate of the control point of the second Bézier curve. i,n (t) is a Bernstein polynomial.

[0114] In one possible implementation, the device further includes:

[0115] The third acquisition module is used to acquire the dialogue methods of the characters in the dialogue, including direct dialogue and remote dialogue.

[0116] The first transmission module is used to transmit two-dimensional animated vector graphics to the vehicle's display screen for display via either the vehicle's internal wireless network or Bluetooth, based on a direct dialogue method.

[0117] The second transmission module is used for remote dialogue, transmitting two-dimensional animated vector graphics to the vehicle's display screen via Bluetooth inside the vehicle for display.

[0118] This device acquires video images of the faces and body parts of people engaged in conversation inside a vehicle. It then processes these images using a target face recognition model and a target body recognition model to obtain facial and body feature points. A 2D animation vector diagram is then created. Based on the coordinates of the facial and body feature points, the coordinates of reference points for the face and body parts within the 2D animation vector diagram are determined. Finally, based on these coordinates, the facial and body contour curves are determined within the 2D animation vector diagram, thus defining the 2D animation of the person engaging in conversation. Because 2D animation has a smaller data volume than video, it reduces the requirements for the memory, transmission capacity, and transmission costs of the in-vehicle equipment.

[0119] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0120] Figure 9 This is a schematic diagram of a server structure provided in an embodiment of this application. The server can vary significantly due to differences in configuration or performance. It may include one or more processors 901 and one or more memories 902. The one or more memories 902 store at least one computer program, which is loaded and executed by the one or more processors 901 to enable the server to implement the dynamic human image generation method provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0121] Figure 10This is a schematic diagram of a dynamic human image generation device according to an embodiment of this application. The device can be a terminal, such as an in-vehicle terminal, smartphone, tablet computer, media player, laptop computer, or desktop computer. The terminal may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0122] Typically, a terminal includes a processor 1501 and a memory 1502.

[0123] Processor 1501 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1501 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1501 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1501 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1501 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0124] The memory 1502 may include one or more computer-readable storage media, which may be non-transitory. The memory 1502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1502 is used to store at least one instruction, which is executed by the processor 1501 to cause the terminal to implement the dynamic human image generation method provided in the method embodiments of this application.

[0125] In some embodiments, the terminal may also optionally include: a peripheral device interface 1503 and at least one peripheral device. The processor 1501, memory 1502, and peripheral device interface 1503 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1503 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1504, a display screen 1505, a camera assembly 1506, an audio circuit 1507, and a power supply 1508.

[0126] Peripheral interface 1503 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1501 and memory 1502. In some embodiments, processor 1501, memory 1502 and peripheral interface 1503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1501, memory 1502 and peripheral interface 1503 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0127] The radio frequency (RF) circuit 1504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1504 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1504 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1504 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0128] Display screen 1505 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1505 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1501 for processing. In this case, display screen 1505 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 1505 can be a single screen, located on the front panel of the terminal; in other embodiments, display screen 1505 can be at least two screens, respectively located on different surfaces of the terminal or in a folded design; in other embodiments, display screen 1505 can be a flexible display screen, located on a curved or folded surface of the terminal. Furthermore, display screen 1505 can be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 1505 can be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0129] The camera assembly 1506 is used to acquire images or videos. Optionally, the camera assembly 1506 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1506 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0130] The audio circuit 1507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1501 for processing, or input to the radio frequency circuit 1504 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1501 or the radio frequency circuit 1504 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1507 may also include a headphone jack.

[0131] Power supply 1508 is used to power the various components in the terminal. Power supply 1508 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1508 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0132] In some embodiments, the terminal further includes one or more sensors 1509. The one or more sensors 1509 include, but are not limited to: an acceleration sensor 1510, a gyroscope sensor 1511, a pressure sensor 1512, an optical sensor 1513, and a proximity sensor 1514.

[0133] Accelerometer 1510 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by the terminal. For example, accelerometer 1510 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1501 can control display screen 1505 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1510. Accelerometer 1510 can also be used for games or for acquiring user motion data.

[0134] The gyroscope sensor 1511 can detect the terminal's orientation and rotation angle. The gyroscope sensor 1511 can work in conjunction with the accelerometer sensor 1510 to collect the user's 3D movements on the terminal. Based on the data collected by the gyroscope sensor 1511, the processor 1501 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0135] The pressure sensor 1512 can be disposed on the side bezel of the terminal and / or the lower layer of the display screen 1505. When the pressure sensor 1512 is disposed on the side bezel of the terminal, it can detect the user's grip signal on the terminal, and the processor 1501 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1512. When the pressure sensor 1512 is disposed on the lower layer of the display screen 1505, the processor 1501 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1505. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0136] Optical sensor 1513 is used to collect ambient light intensity. In one embodiment, processor 1501 can control the display brightness of display screen 1505 based on the ambient light intensity collected by optical sensor 1513. Specifically, when the ambient light intensity is high, the display brightness of display screen 1505 is increased; when the ambient light intensity is low, the display brightness of display screen 1505 is decreased. In another embodiment, processor 1501 can also dynamically adjust the shooting parameters of camera assembly 1506 based on the ambient light intensity collected by optical sensor 1513.

[0137] The proximity sensor 1514, also known as a distance sensor, is typically installed on the front panel of the terminal. The proximity sensor 1514 is used to detect the distance between the user and the front of the terminal. In one embodiment, when the proximity sensor 1514 detects that the distance between the user and the front of the terminal is gradually decreasing, the processor 1501 controls the display screen 1505 to switch from a screen-on state to a screen-off state; when the proximity sensor 1514 detects that the distance between the user and the front of the terminal is gradually increasing, the processor 1501 controls the display screen 1505 to switch from a screen-off state to a screen-on state.

[0138] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0139] In an exemplary embodiment, a dynamic portrait generation apparatus is also provided. The dynamic portrait generation apparatus includes a processor and a memory, the memory storing at least one computer program. The at least one computer program is loaded and executed by one or more processors to enable the dynamic portrait generation apparatus to implement any of the aforementioned dynamic portrait generation methods.

[0140] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor of a dynamic human image generation device to enable a computer to implement any of the above-described methods for generating dynamic human images.

[0141] In one possible implementation, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0142] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of the dynamic portrait generation device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the dynamic portrait generation device to perform any of the aforementioned dynamic portrait generation methods.

[0143] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the video images of the faces and body parts of the dialogue participants involved in this application were obtained with full authorization.

[0144] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0145] It should be noted that the terms "first," "second," etc. (if applicable) in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0146] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for generating dynamic human portraits, characterized in that, The method includes: The system acquires video images of people conversing inside the vehicle, including video images of the faces and body parts of the people conversing. The video image of the face is processed by the target face recognition model to obtain a first number of facial feature points, and the video image of the body part is processed by the target body recognition model to obtain a second number of body part feature points. Based on the position coordinates of the first number of facial feature points, determine the position coordinates of the first number of facial reference points in the two-dimensional animation vector image; based on the position coordinates of the second number of body part feature points, determine the position coordinates of the second number of body part reference points in the two-dimensional animation vector image. Substitute the position coordinates of the first number of facial reference points into the function of the first Bézier curve to obtain the control point coordinates of the first Bézier curve; Based on the control point coordinates of the first Bézier curve, the first Bézier curve is determined and used as the facial contour curve in the two-dimensional animation vector image. The function of the first Bézier curve is: The x(t) is a function of the first Bézier curve, where t is the independent variable of the function of the first Bézier curve, x(t) is the abscissa of the facial reference point, and y(t) is the ordinate of the facial reference point. Let x be the x-coordinate of the control point of the first Bézier curve. The ordinate of the control point of the first Bézier curve is given. For Bernstein polynomials; Based on the position coordinates of the second number of body part reference points, determine the body part outline curve in the two-dimensional animation vector image; The 2D animation of the dialogue character is determined based on the facial contour curve and the body part contour curve in the 2D animation vector image.

2. The method according to claim 1, characterized in that, The method further includes: Obtain a facial training image and a corresponding facial reference image, wherein the facial reference image includes a first number of labeled facial feature points; Based on the initial face recognition model, a first number of facial feature points are determined on the facial training image; A loss value is determined based on the facial feature points marked on the facial reference image and the facial feature points on the facial training image. The initial face recognition model is then adjusted based on the loss value to obtain the target face recognition model.

3. The method according to claim 1, characterized in that, The method further includes: Acquire a training image of a body part and a reference image of the body part corresponding to the training image of the body part, wherein the reference image of the body part includes a second number of labeled body part feature points; Based on the initial body recognition model, a second number of body part feature points are determined on the training image of the body part; The loss value is determined based on the body part feature points marked on the body part reference image and the body part feature points on the body part training image. The initial body recognition model is adjusted based on the loss value to obtain the target body recognition model.

4. The method according to claim 1, characterized in that, The step of determining the body part contour curve in the two-dimensional animation vector image based on the position coordinates of the second number of body part reference points includes: Substitute the position coordinates of the second number of body part reference points into the function of the second Bézier curve to obtain the control point coordinates of the second Bézier curve; Based on the control point coordinates of the second Bézier curve, the second Bézier curve is determined and used as the outline curve of the body part in the two-dimensional animation vector image. The function of the second Bézier curve is: The x(t) is a function of the second Bézier curve, where t is the independent variable of the function of the second Bézier curve, x(t) is the abscissa of the body part reference point, and y(t) is the ordinate of the body part reference point. The x-coordinate of the control point of the second Bézier curve, The ordinate of the control point of the second Bézier curve is given. It is a Bernstein polynomial.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Obtain the dialogue mode of the characters in the dialogue, including direct dialogue and remote dialogue; Based on the fact that the dialogue method is direct dialogue, the two-dimensional animated vector graphic is transmitted to the vehicle's display screen for display via either the vehicle's internal wireless network or Bluetooth transmission. Based on the remote dialogue method, the two-dimensional animated vector graphic is transmitted to the vehicle's display screen via Bluetooth inside the vehicle for display.

6. A device for generating dynamic human images, characterized in that, The device includes: The acquisition module is used to acquire video images of people conversing inside the vehicle. The video images include video images of the faces of the people conversing and video images of the body parts of the people conversing. The recognition module is used to perform facial recognition processing on the video image of the face through the target face recognition model to obtain a first number of facial feature points, and to perform body part recognition processing on the video image of the body part through the target body recognition model to obtain a second number of body part feature points. The first determining module is used to determine the position coordinates of a first number of facial reference points in the two-dimensional animation vector image based on the position coordinates of the first number of facial feature points, and to determine the position coordinates of a second number of body part reference points in the two-dimensional animation vector image based on the position coordinates of the second number of body part feature points. The second determining module is used to determine the facial contour curve in the two-dimensional animation vector image based on the position coordinates of the first number of facial reference points, and to determine the body part contour curve in the two-dimensional animation vector image based on the position coordinates of the second number of body part reference points. The third determining module is used to determine the two-dimensional animation of the dialogue character based on the facial contour curve and the body part contour curve in the two-dimensional animation vector image.

7. The apparatus according to claim 6, characterized in that, The device further includes: The first acquisition module is used to acquire a facial training image and a facial reference image corresponding to the facial training image, wherein the facial reference image includes a first number of labeled facial feature points; The fourth determining module is used to determine a first number of facial feature points on the facial training image based on the initial face recognition model; The first adjustment module is used to determine a loss value based on the facial feature points marked on the facial reference image and the facial feature points on the facial training image, and adjust the initial face recognition model based on the loss value to obtain the target face recognition model.

8. A device for generating dynamic human images, characterized in that, The dynamic portrait generation device includes a processor and a memory, wherein the memory stores at least one computer program, which is loaded and executed by the processor to enable the dynamic portrait generation device to implement the dynamic portrait generation method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer to implement the method for generating dynamic human figures as described in any one of claims 1 to 5.

Images