Method and device for determining target person, terminal equipment and computer program product

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a target person determination method and device, terminal equipment and a computer program product, and the method comprises the steps: determining n to-be-detected persons in a display image at a kth moment; acquiring speaking feature data related to the n to-be-tested persons from the display picture; and determining a target person in the display picture at the kth moment from the n to-be-detected persons based on the speaking feature data. According to the target person determination method provided by the invention, the n to-be-detected persons in the display picture at the kth moment are determined, then the speaking feature data related to the n to-be-detected persons are acquired from the display picture, and the target person in the display picture at the kth moment is determined from the n to-be-detected persons based on the speaking feature data. The target person can be accurately and efficiently determined from the multiple person pictures at low cost.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to a method, apparatus, terminal device and computer program product for determining a target person. Background Art

[0002] With the development of society and the advancement of technology, today's conference rooms are not only equipped with larger conference display screens, but also with conference cameras to facilitate remote video conferencing.

[0003] In actual work and life, remote meetings often involve multiple people. In order to improve the meeting experience and efficiency, it is necessary to accurately identify the speaker among the multiple participants. The existing technology generally uses sound source localization technology to determine the location of the sound source and then the location of the speaker, so as to determine the speaker from the picture scene of multiple characters. However, sound source localization has many disadvantages. First, it has high requirements for hardware and requires a microphone array to achieve, which is costly. Second, in noisy environments, such as family meetings, sound source localization is easily affected by echoes or noise, resulting in poor accuracy, thereby affecting the accuracy of determining the speaker from multiple characters.

[0004] Therefore, an accurate and low-cost method is needed to confirm the speaker from a scene with multiple characters. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a method, apparatus, terminal device and computer program product for determining a target person, so as to achieve low-cost, accurate and efficient determination of a target person from multiple character images.

[0006] A first aspect of an embodiment of the present application provides a method for determining a target person, including:

[0007] Determine n persons to be tested in the display screen at the kth moment, where k and n are both positive integers;

[0008] Acquiring speech feature data related to the n persons to be tested from the display screen;

[0009] The target person in the display image at the kth moment is determined from the n persons to be tested based on the speech feature data.

[0010] In an implementation of the first aspect, acquiring speech feature data related to the n persons to be tested from the display screen includes:

[0011] Respectively obtain the mouth feature points of each of the persons to be tested in the display image;

[0012] Calculating the mouth shape value of the person to be tested based on the mouth feature points, and storing the mouth shape value of the person to be tested on the display screen in a cache area corresponding to the person to be tested;

[0013] The number of speaking actions of each person to be tested is calculated based on the historical mouth shape values ​​in the cache area, and the candidate person is determined according to the number of speaking actions; wherein the historical mouth shape values ​​are the mouth shape values ​​of each person to be tested determined by the display screen of multiple consecutive frames before the display screen at the kth moment;

[0014] The step of determining the target person in the display image at the kth moment from among the n persons to be tested based on the speech feature data comprises:

[0015] A target person is determined from among the candidate persons based on the number of speaking actions.

[0016] In an implementation of the first aspect, the calculating the number of speaking actions of each person to be tested based on the historical mouth shape values ​​in the cache area, and determining the candidate person according to the number of speaking actions includes:

[0017] For each target person to be tested among the n persons to be tested, if the number of historical mouth shape values ​​in the cache area of ​​the target person to be tested reaches a first preset threshold, the following steps are performed:

[0018] The historical mouth shape first stored in the cache area is set as the initial mouth shape value;

[0019] Set the initial mouth shape value to the maximum mouth shape value and the minimum mouth shape value;

[0020] Traversing the historical mouth shape values ​​in the cache area starting from the initial mouth shape value;

[0021] If the traversed target historical mouth shape value is greater than the maximum mouth shape value, the target historical mouth shape value is set as the new maximum mouth shape value;

[0022] If the traversed target historical mouth shape value is less than the minimum mouth shape value, the target historical mouth shape value is set as a new minimum mouth shape value;

[0023] If the traversed target historical mouth shape value is not less than the minimum mouth shape value, not greater than the maximum mouth shape value, and the difference between the maximum mouth shape value and the minimum mouth shape value is not less than the preset mouth shape difference threshold, then the number of speaking actions of the target person to be tested is accumulated, the target historical mouth shape value is set as a new initial mouth shape value, and the step of traversing the historical mouth shape values ​​in the cache area starting from the initial mouth shape value is returned;

[0024] When the traversal of the historical mouth shape values ​​in the cache area is completed, if the number of speaking actions is greater than a preset speaking determination threshold, the target person to be tested is determined as a candidate person.

[0025] In an implementation of the first aspect, determining a target person from each of the candidate persons based on the number of speaking actions includes:

[0026] The speaking action times of the candidate characters are compared, and the candidate character with the largest speaking action time is selected as the target character in the display picture at the kth moment.

[0027] In an implementation of the first aspect, determining n persons to be tested in the display picture at the kth moment includes:

[0028] Recognize the face information in the display picture at the kth moment;

[0029] Determine n persons to be tested from each piece of facial information, wherein the facial state of the persons to be tested is a frontal face state.

[0030] In an implementation of the first aspect, the method further includes:

[0031] The target person is highlighted in the display picture at the kth moment in a first preset manner.

[0032] In an implementation of the first aspect, the method further includes:

[0033] Obtaining the target person in the display image at the kth moment;

[0034] Determine m persons to be tested in the display screen at the k+Lth time, where the k+Lth time represents a future time;

[0035] Acquire new speech feature data related to the m persons to be tested from the display picture at the k+Lth moment;

[0036] Determine a new target person in the display screen at the k+Lth moment from the m persons to be tested based on the new speech feature data;

[0037] Acquire a new target person in the display image at the k+Lth moment;

[0038] In the case that the target person is different from the new target person, the new target person is highlighted in the display picture at the k+Lth moment according to a second preset manner.

[0039] In an implementation of the first aspect, the highlighting of the target person in the display picture at the kth moment in a first preset manner includes:

[0040] The target person is used as the display content of the display screen at the kth moment.

[0041] In an implementation of the first aspect, when the target person is different from the new target person, highlighting the new target person in the display screen at the k+Lth time in a second preset manner includes:

[0042] In a case where the target person is different from the new target person, the current display content of the display screen is switched from the target person to the new target person.

[0043] In an implementation of the first aspect, when the target person and the new target person are different, switching the current display content of the display screen from the target person to the new target person includes:

[0044] When the target person is different from the new target person, the size and position of the display content in the display screen are uniformly changed based on the size information and position information of the target person and the size information and position information of the new target person, until the display content in the display screen at the k+Lth moment is the new target person.

[0045] A second aspect of an embodiment of the present application provides a device for determining a target person, including:

[0046] A first determining unit is used to determine n persons to be detected in the display picture at the kth moment, where k and n are both positive integers;

[0047] An acquisition unit, used for acquiring speech feature data related to the n persons to be tested from the display screen;

[0048] The second determining unit is used to determine the target person in the display image at the kth moment from the n persons to be tested based on the speech feature data.

[0049] A third aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the method for determining a target person as described in the first aspect are implemented.

[0050] A fourth aspect of an embodiment of the present application provides a computer program product, including a computer program, and when the computer program is run, the method for determining a target person as described in the first aspect is executed.

[0051] The beneficial effect of the first aspect of the embodiment of the present application is: by determining n persons to be tested in the display screen at the kth moment, and then obtaining speech feature data related to the n persons to be tested from the display screen, based on the speech feature data, determining the target person in the display screen at the kth moment from the n persons to be tested, thereby achieving low-cost, accurate and efficient determination of the target person from multiple character screens.

[0052] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 This is a schematic diagram of an implementation flow of a method for determining a target person provided in an embodiment of the present application;

[0055] Figure 2 is a schematic diagram of a mouth feature point provided in an embodiment of the present application;

[0056] Figure 3 is a schematic diagram of mouth feature points involved in mouth shape value calculation provided in an embodiment of the present application;

[0057] Figure 4 This is a schematic diagram of a mouth shape value cache structure provided in an embodiment of the present application;

[0058] Figure 5 It is a flowchart of calculating the number of speaking actions and determining candidate characters in a method for determining a target character provided in an embodiment of the present application;

[0059] Figure 6 This is a schematic diagram of the implementation flow of the second method for determining a target person provided in an embodiment of the present application;

[0060] Figure 7 is a schematic diagram of a device for determining a target person provided in an embodiment of the present application;

[0061] Figure 8 is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0062] Fig. 9 It is a schematic diagram of a computer program product provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0064] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0065] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0066] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0067] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0068] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0069] The embodiment of the present application provides a method for determining a target person, which is used to determine the target person from multiple character images. The method for determining a target person provided by the embodiment of the present application determines n persons to be tested in a display image at the kth moment, then obtains speech feature data related to the n persons to be tested from the display image, and determines the target person in the display image at the kth moment from the n persons to be tested based on the speech feature data, thereby realizing low-cost, accurate and efficient determination of the target person from multiple character images.

[0070] The target person determination method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), desktop computers, televisions, smart display screens, etc. The embodiment of the present application does not impose any restrictions on the specific type of terminal devices.

[0071] like Figure 1 As shown, the embodiment of the present application provides a method for determining a target person, including steps S10 to S30:

[0072] Step S10, determining n persons to be tested in the display image at the kth moment, where k and n are both positive integers.

[0073] In the application, the display screen includes image data collected by the camera and then displayed on the terminal device. The person to be tested is a person in the display screen who may be the target person. There are generally multiple people to be tested in the display screen at the kth moment, and the number of people to be tested in the display screen at the kth moment is generally less than the number of all people in the display screen at the kth moment. Taking the conference scene as an example, the n people to be tested in the display screen at the kth moment are the participants who may be the speaker at the kth moment.

[0074] In one embodiment, step S10 includes:

[0075] Step S11, identifying the face information in the display image at the kth moment.

[0076] Step S12, determining n persons to be tested from each facial information, wherein the facial state of the persons to be tested is a frontal face state.

[0077] In the application, the face information (face bounding box) in the display image at the kth moment is identified based on the face detection model. Each face frame includes the face information of a person and defines the position and size of the included face information. Each face bounding box is cropped after expansion according to a certain ratio and can be used for subsequent processing. It can be understood that in order to ensure the integrity of the face information, the actual cropped area will be slightly larger than the detected face border.

[0078] In the application, the target person will face the camera that collects the image data in the display screen. Therefore, by judging whether the state of the face in the face information in the display screen is in the front face state, the interfering person who is not the target person can be excluded, leaving only the person to be tested whose face state is in the front face state. For example, in a meeting scenario, there are generally several or even dozens of people participating in the meeting, and there is only one speaker at the kth moment. If a target person is to be determined among dozens of people in the display screen, the data processing volume is large and there are more interferences. Therefore, the method for determining the target person provided in the embodiment of the present application obtains the face information in the display screen at the kth moment, and then determines that the person whose face information is in the front face state is the person to be tested, thereby narrowing the confirmation range of the target person and improving the confirmation efficiency and accuracy of the target person.

[0079] Step S20, acquiring speech feature data related to the n persons to be tested from the display screen.

[0080] In the application, the speech feature data related to n persons to be tested can be obtained by performing image processing on multiple frames of display images before the kth moment. Since the number of persons to be tested in the display image is generally less than the number of all persons in the display image, by only obtaining speech feature data related to the persons to be tested, the resource occupation of obtaining speech feature data related to non-persons to be tested can be reduced, thereby improving the efficiency of image processing.

[0081] Step S30 , determining the target person in the display image at the kth moment from among the n persons to be tested based on the speech feature data.

[0082] In the application, the target person in the display may also change over time. For example, in a meeting scenario, the speaker may change as the meeting progresses and the current topic of the meeting changes. It is understandable that the target person's speech feature data is different from other test persons. For example, in a meeting scenario, the target person is the speaker, and the speaker's speech feature data, such as speaking duration, speaking continuity, and speaking frequency, are different from those of other participants.

[0083] In one embodiment, the step S20, obtaining speech feature data related to the n persons to be tested from the display screen, includes:

[0084] Step S21, respectively obtaining the mouth feature points of each of the persons to be tested in the display image.

[0085] In the application, in the meeting scene, the face state of the person to be tested is the front face state. The expression recognition model can be used to obtain the expression of each person to be tested in the display screen. Figure 2 The multiple mouth feature points shown in FIG. 1 include at least Figure 3 The four feature points shown are: feature point A in the upper middle of the lips, feature point B in the lower middle of the lips, feature point C at the left corner of the mouth, and feature point D at the right corner of the mouth.

[0086] Step S22, calculating the mouth shape value of the person to be tested based on the mouth feature points, and storing the mouth shape value of the person to be tested in the display screen into a cache area corresponding to the person to be tested.

[0087] In the application, based on the mouth feature points including the upper-middle feature point A of the lips, the lower-middle feature point B of the lips, the left corner of the mouth feature point C and the right corner of the mouth feature point D, the mouth shape value of the person to be tested is calculated through the preset mouth shape recognition algorithm. The specific calculation process is as follows:

[0088] Assuming the coordinates of the feature points are A(x1, y1), B(x2, y2), C(x3, y3), and D(x4, y4), the mouth opening and closing distance is

[0089] Mouth length distance

[0090] The ratio of the mouth opening and closing distance (mouth shape value) Rate = D1 / D2.

[0091] In the application, the above algorithm can be used to calculate the mouth shape value of the person to be tested, and then the mouth shape value of the person to be tested in the display screen is stored in the cache area corresponding to the person to be tested. Each person to be tested corresponds to a cache area.

[0092] In the application, each person to be tested corresponds to a mouth shape value buffer area (buffer). Each mouth shape value buffer area can store up to N mouth shape values. Each time it is saved, assuming that the data to be stored is Rate, when the address to be saved is n (the initial value of n is 0), the mouth shape value is stored in the mouth shape value buffer area through buffer[n]=Rate; after each successful save, n needs to be moved back one position, that is, n=n+1; in order to prevent n from exceeding the maximum buffer quantity boundary N, it is necessary to go back to the beginning to cover the farthest historical data when it exceeds, so if n>=N, then n=0; in this way, the mouth shape value buffer area buffer works in a circular cache mode. The structure of the mouth shape value buffer can be referred to Figure 4 shown.

[0093] Step S23, calculating the number of speaking actions of each person to be tested based on the historical mouth shape values ​​in the cache area, and determining the candidate person according to the number of speaking actions; wherein the historical mouth shape values ​​are the mouth shape values ​​of each person to be tested determined by multiple consecutive frames of display images before the display image at the kth moment.

[0094] In the application, the number of speaking actions is calculated based on the historical mouth shape values. For example, yawning or eating may also cause the mouth shape to change similar to speaking. Therefore, it is necessary to further determine the candidate person among the persons to be tested based on the number of speaking actions to eliminate the above interference. It can be understood that the probability that the candidate person is the target person is greater than the probability that the person to be tested is the target person. By confirming the candidate person among the persons to be tested, the efficiency and accuracy of determining the target person are improved.

[0095] The step S30, determining the target person in the display image at the kth moment from the n persons to be tested based on the speech feature data, comprises:

[0096] Step S31, determining a target person from each of the candidate persons based on the number of speaking actions.

[0097] In the application, the candidate person has a preset number of speaking actions. For different target persons in different scenarios, corresponding target person determination conditions can be set to determine the target person from the candidate person based on the number of speaking actions. In a meeting scenario, in addition to the target person (speaker) speaking, there may also be people communicating among the participants. Therefore, the candidate person includes the target person and other persons to be tested who may be the target person and are speaking.

[0098] like Figure 5 As shown, in one embodiment, the step S23, calculating the number of speaking actions of each person to be tested based on the historical mouth shape values ​​in the cache area, and determining the candidate person according to the number of speaking actions, includes:

[0099] Step S231: for each target person to be tested among the n persons to be tested, if the number of historical mouth shape values ​​in the cache area reaches a first preset threshold, the following steps are performed:

[0100] Step S232, setting the historical mouth shape first stored in the cache area as the initial mouth shape value.

[0101] Step S233, setting the initial mouth shape value to the maximum mouth shape value and the minimum mouth shape value.

[0102] Step S234, traversing the historical mouth shape values ​​in the cache area starting from the initial mouth shape value.

[0103] Step S235: If the traversed target historical mouth shape value is greater than the maximum mouth shape value, the target historical mouth shape value is set as the new maximum mouth shape value.

[0104] Step S236: If the traversed target historical mouth shape value is less than the minimum mouth shape value, the target historical mouth shape value is set as the new minimum mouth shape value.

[0105] Step S237, if the traversed target historical mouth shape value is not less than the minimum mouth shape value, not greater than the maximum mouth shape value, and the difference between the maximum mouth shape value and the minimum mouth shape value is not less than the preset mouth shape difference threshold, then the number of speaking actions of the target person to be tested is accumulated, the target historical mouth shape value is set as a new initial mouth shape value, and the process returns to step S234, starting from the initial mouth shape value to traverse the historical mouth shape values ​​in the cache area.

[0106] Step S238, when the traversal of each historical mouth shape value in the cache area is completed, if the number of speaking actions is greater than a preset speaking determination threshold, the target person to be tested is determined as a candidate person.

[0107] In the application, the change range of the mouth shape value varies in different languages, so the preset mouth shape difference threshold can be determined according to the language type in different scenarios.

[0108] In the application, the speech determination threshold can be set according to the actual scenario of determining the target person, and the speech determination threshold can be increased or decreased according to different needs. The larger the speech determination threshold, the lower the proportion of candidate persons screened from the person to be tested. Through the above steps S231 to S238, the number of speaking actions of each person to be tested can be accurately calculated based on the historical mouth shape value, so as to determine the candidate person from the person to be tested, further narrow the scope of determining the target person, and achieve more efficient and accurate determination of the target person.

[0109] In one embodiment, the step S31, determining the target person from each of the candidate persons based on the number of speaking actions, includes:

[0110] Step S311 , comparing the speaking action times of the candidate characters, and selecting the candidate character with the largest speaking action time as the target character in the display screen at the kth moment.

[0111] In the application, the target character's speaking action count is the largest among the candidate characters. For example, in a meeting scenario, the target character, as the speaker, will speak in the meeting. Even if other participants speak privately, their speaking times will not exceed the speaker's.

[0112] like Figure 6As shown, in one embodiment, the method further includes:

[0113] Step S40: highlighting the target person in the display image at the kth moment in a first preset manner.

[0114] In the application, after the target person is determined, the target person can be highlighted in the display screen at the kth moment. For example, in a meeting scenario, the speaker can be enlarged to focus on the speaker so that participants can better pay attention to the speaker's speech screen.

[0115] In one embodiment, the method further comprises:

[0116] Step S50, obtaining the target person in the display image at the kth moment.

[0117] Step S60, determining m persons to be tested in the display image at the k+Lth time, where the k+Lth time represents a future time.

[0118] Step S70, acquiring new speech feature data related to the m persons to be tested from the display picture at the k+Lth moment.

[0119] Step S80: determining a new target person in the display image at the k+Lth moment from among the m persons to be tested based on the new speech feature data.

[0120] Step S90, obtaining a new target person in the display image at the k+Lth moment.

[0121] Step S100: When the target person is different from the new target person, the new target person is highlighted in the display screen at the k+Lth moment according to a second preset manner.

[0122] In the application, taking the conference scene as an example, the speaker may change as the conference progresses and the sub-topics of the conference change. Therefore, if the new target person in the display screen at the k+Lth moment is different from the target person in the display screen at the kth moment, it means that the speaker has changed, and the new target person needs to be highlighted in the display screen at the k+Lth moment according to the second preset method.

[0123] In one embodiment, the step S40, highlighting the target person in the display picture at the kth moment in a first preset manner, includes:

[0124] Step S41: using the target person as the display content of the display screen at the kth moment.

[0125] In the application, after the target person is determined, the target person can be marked out from all the characters in the display screen by a prominent display frame in the display screen at the kth moment, thereby achieving a complete display of multiple character scenes while highlighting the target person.

[0126] In the application, after the target person is determined, the target person can also be highlighted separately in the display screen at the kth moment, that is, the target person is used as the only display content of the display screen at the kth moment. For example, in a conference scene, by highlighting the speaker separately, a better conference focus effect is achieved, so that participants can focus more on the speaker.

[0127] In one embodiment, in step S100, when the target person is different from the new target person, highlighting the new target person in the display screen at the k+Lth time in a second preset manner includes:

[0128] Step S1001 , when the target person is different from the new target person, the current display content of the display screen is switched from the target person to the new target person.

[0129] In the application, when the target person is different from the new target person, the current display content of the display screen is switched to the new target person, so that the display content of the display screen is updated as the target person changes.

[0130] In the application, when it is detected that the target person determined at the kth moment is different from the new target person determined at the k+Lth moment, the currently displayed content is not immediately switched from the target person to the new target person, but is delayed for a certain time t according to a preset delay method, and then the currently displayed content is switched from the target person to the new target person; the above method avoids frequent switching of the display content of the display screen, improves the stability of the display screen, and optimizes the viewing experience of the display screen.

[0131] In one embodiment, when the target person and the new target person are different, switching the current display content of the display screen from the target person to the new target person includes:

[0132] Step S1002, when the target person is different from the new target person, based on the size information and position information of the target person and the size information and position information of the new target person, the size and position of the display content in the display screen are uniformly changed until the display content in the display screen at the k+Lth moment is the new target person.

[0133] In the application, when the target person is highlighted separately in the display screen, it is achieved by cutting and scaling the original image of the display screen including multiple characters. In order to avoid sudden changes in the content displayed in the display screen, the current display content of the display screen is switched from the target person to the new target person by uniformly transforming the size and position of the display content in the display screen based on the size information of the target person, until the display content in the display screen at the k+Lth moment is the new target person. Specifically, the change range of the size and position of the display content in the display screen can be calculated based on the size information of the target person displayed in the display screen at the kth moment and the position information of the target person in the display screen including multiple characters, as well as the size of the new target person and the position information of the new target person in the display screen including multiple characters, and then the size and position of the display content in the display screen are uniformly transformed frame by frame, so that the display content in the display screen at the k+Lth moment is the new target person. It can be understood that, through the above method, when the target person and the new target person are different, the smooth transition of the display content in the display screen is achieved, and the display effect is optimized.

[0134] In application, the target person determination method provided in this application can assist the use of existing sound source localization technology. In a conference scenario, the speaker recognition result and the sound source localization result can be combined to improve the accuracy of speaker positioning; or when the sound source positioning is interfered by noise, the target person determination method provided in this solution can be used to filter out the sound source position of the person who does not have face data or is in a frontal face state, thereby achieving more accurate positioning.

[0135] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0136] The present application also provides a device for determining a target person, which is used to perform the steps in the above-mentioned method for determining a target person. The device for determining a target person can be a virtual appliance in a terminal device, which is run by a processor of the first terminal device, or it can be the terminal device itself.

[0137] like Figure 7 As shown, the target person determination device 7 provided in the embodiment of the present application includes:

[0138] The first determining unit 701 is used to determine n persons to be detected in the display picture at the kth moment, where k and n are both positive integers;

[0139] An acquisition unit 702, configured to acquire speech feature data related to the n persons to be tested from the display screen;

[0140] The second determining unit 703 is configured to determine the target person in the display image at the kth moment from among the n persons to be tested based on the speech feature data.

[0141] In one embodiment, the acquisition unit 702 is used to: respectively acquire the mouth feature points of each of the persons to be tested in the display image;

[0142] Calculating the mouth shape value of the person to be tested based on the mouth feature points, and storing the mouth shape value of the person to be tested on the display screen in a cache area corresponding to the person to be tested;

[0143] The number of speaking actions of each person to be tested is calculated based on the historical mouth shape values ​​in the cache area, and the candidate person is determined according to the number of speaking actions; wherein the historical mouth shape values ​​are the mouth shape values ​​of each person to be tested determined by the display screen of multiple consecutive frames before the display screen at the kth moment;

[0144] The second determining unit 703 is configured to determine a target person from among the candidate persons based on the number of speaking actions.

[0145] In one embodiment, the acquisition unit 702 is used to: for each target person to be tested among the n persons to be tested, if the number of historical mouth shape values ​​in the cache area reaches a first preset threshold, perform the following steps:

[0146] The historical mouth shape first stored in the cache area is set as the initial mouth shape value;

[0147] Set the initial mouth shape value to the maximum mouth shape value and the minimum mouth shape value;

[0148] Traversing the historical mouth shape values ​​in the cache area starting from the initial mouth shape value;

[0149] If the traversed target historical mouth shape value is greater than the maximum mouth shape value, the target historical mouth shape value is set as the new maximum mouth shape value;

[0150] If the traversed target historical mouth shape value is less than the minimum mouth shape value, the target historical mouth shape value is set as a new minimum mouth shape value;

[0151] If the traversed target historical mouth shape value is not less than the minimum mouth shape value, not greater than the maximum mouth shape value, and the difference between the maximum mouth shape value and the minimum mouth shape value is not less than the preset mouth shape difference threshold, then the number of speaking actions of the target person to be tested is accumulated, the target historical mouth shape value is set as a new initial mouth shape value, and the step of traversing the historical mouth shape values ​​in the cache area starting from the initial mouth shape value is returned;

[0152] When the traversal of the historical mouth shape values ​​in the cache area is completed, if the number of speaking actions is greater than a preset speaking determination threshold, the target person to be tested is determined as a candidate person.

[0153] In one embodiment, the acquisition unit 702 is used to compare the speaking action times of each candidate character, and select the candidate character with the largest speaking action time as the target character in the display picture at the kth moment.

[0154] In one embodiment, the first determining unit 701 is used to identify face information in the display picture at the kth moment;

[0155] Determine n persons to be tested from each piece of facial information, wherein the facial state of the persons to be tested is a frontal face state.

[0156] In one embodiment, the target person determination device 7 further includes:

[0157] The display unit 704 is used to highlight the target person in the display picture at the kth moment in a first preset manner.

[0158] In one embodiment, the display unit 704 is further used to: obtain the target person in the display picture at the kth moment;

[0159] Determine m persons to be tested in the display screen at the k+Lth time, where the k+Lth time represents a future time;

[0160] Acquire new speech feature data related to the m persons to be tested from the display picture at the k+Lth moment;

[0161] Determine a new target person in the display screen at the k+Lth moment from the m persons to be tested based on the new speech feature data;

[0162] Acquire a new target person in the display image at the k+Lth moment;

[0163] In the case that the target person is different from the new target person, the new target person is highlighted in the display picture at the k+Lth moment according to a second preset manner.

[0164] In one embodiment, the display unit 704 is used to use the target person as the display content of the display screen at the kth moment.

[0165] In one embodiment, the display unit 704 is used to switch the current display content of the display screen from the target person to the new target person when the target person is different from the new target person.

[0166] In one embodiment, the display unit 704 is used to uniformly change the size and position of the display content in the display screen based on the size information and position information of the target person and the size information and position information of the new target person when the target person is different from the new target person, until the display content in the display screen at the k+Lth moment is the new target person.

[0167] Figure 8 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application. Figure 8 As shown, the terminal device 8 of this embodiment includes: at least one processor 80 ( Figure 8 Only one is shown in the figure) a processor, a memory 81, and a computer program 82 stored in the memory 81 and executable on the at least one processor 80, wherein the processor 80 implements the steps in any of the above-mentioned target person determination method embodiments when executing the computer program 82.

[0168] The terminal device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will appreciate that Figure 8 It is only an example of the terminal device 8 and does not constitute a limitation on the terminal device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0169] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0170] In some embodiments, the memory 81 may be an internal storage unit of the terminal device 8, such as a hard disk or memory of the terminal device 8. In other embodiments, the memory 81 may also be an external storage device of the terminal device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 8. Further, the memory 81 may also include both an internal storage unit and an external storage device of the terminal device 8. The memory 81 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0171] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0172] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0173] An embodiment of the present application further provides a computer storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0174] like Fig. 9 As shown, an embodiment of the present application provides a computer program product 9, including a computer program 90. When the computer program 90 is executed, the steps in the above-mentioned various method embodiments are executed.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0176] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0177] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0178] In the embodiments provided in the present application, it should be understood that the disclosed terminal devices and methods can be implemented in other ways. For example, the terminal device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for determining a target person, characterized in that: include: Determine n persons to be tested in the display screen at the kth moment, where k and n are both positive integers; Acquiring speech feature data related to the n persons to be tested from the display screen; The target person in the display image at the kth moment is determined from the n persons to be tested based on the speech feature data.

2. The method for determining a target person according to claim 1, wherein: The acquiring speech feature data related to the n persons to be tested from the display screen includes: Respectively obtain the mouth feature points of each of the persons to be tested in the display image; Calculating the mouth shape value of the person to be tested based on the mouth feature points, and storing the mouth shape value of the person to be tested on the display screen in a cache area corresponding to the person to be tested; The number of speaking actions of each person to be tested is calculated based on the historical mouth shape values ​​in the cache area, and the candidate person is determined according to the number of speaking actions; wherein the historical mouth shape values ​​are the mouth shape values ​​of each person to be tested determined by the display screen of multiple consecutive frames before the display screen at the kth moment; The step of determining the target person in the display image at the kth moment from among the n persons to be tested based on the speech feature data comprises: A target person is determined from among the candidate persons based on the number of speaking actions.

3. The method for determining a target person according to claim 2, wherein: The step of calculating the number of speaking actions of each person to be tested based on the historical mouth shape values ​​in the cache area, and determining a candidate person according to the number of speaking actions includes: For each target person to be tested among the n persons to be tested, if the number of historical mouth shape values ​​in the cache area of ​​the target person to be tested reaches a first preset threshold, the following steps are performed: The historical mouth shape first stored in the cache area is set as the initial mouth shape value; Set the initial mouth shape value to the maximum mouth shape value and the minimum mouth shape value; Traversing the historical mouth shape values ​​in the cache area starting from the initial mouth shape value; If the traversed target historical mouth shape value is greater than the maximum mouth shape value, the target historical mouth shape value is set as the new maximum mouth shape value; If the traversed target historical mouth shape value is less than the minimum mouth shape value, the target historical mouth shape value is set as a new minimum mouth shape value; If the traversed target historical mouth shape value is not less than the minimum mouth shape value, not greater than the maximum mouth shape value, and the difference between the maximum mouth shape value and the minimum mouth shape value is not less than the preset mouth shape difference threshold, then the number of speaking actions of the target person to be tested is accumulated, the target historical mouth shape value is set as a new initial mouth shape value, and the step of traversing the historical mouth shape values ​​in the cache area starting from the initial mouth shape value is returned; When the traversal of the historical mouth shape values ​​in the cache area is completed, if the number of speaking actions is greater than a preset speaking determination threshold, the target person to be tested is determined as a candidate person.

4. The method for determining a target person according to claim 3, wherein: The step of determining a target person from among the candidate persons based on the number of speaking actions includes: The speaking action times of the candidate characters are compared, and the candidate character with the largest speaking action time is selected as the target character in the display picture at the kth moment.

5. The method for determining a target person according to any one of claims 1 to 4, characterized in that: The step of determining n persons to be tested in the display screen at the kth moment includes: Recognize the face information in the display picture at the kth moment; Determine n persons to be tested from each piece of facial information, wherein the facial state of the persons to be tested is a frontal face state.

6. The method for determining a target person according to any one of claims 1 to 4, characterized in that: The method further comprises: The target person is highlighted in the display picture at the kth moment in a first preset manner.

7. The method for determining a target person according to claim 6, wherein: The method further comprises: Obtaining the target person in the display image at the kth moment; Determine m persons to be tested in the display screen at the k+Lth time, where the k+Lth time represents a future time; Acquire new speech feature data related to the m persons to be tested from the display picture at the k+Lth moment; Determine a new target person in the display screen at the k+Lth moment from the m persons to be tested based on the new speech feature data; Acquire a new target person in the display image at the k+Lth moment; In the case that the target person is different from the new target person, the new target person is highlighted in the display picture at the k+Lth moment according to a second preset manner.

8. The method for determining a target person according to claim 7, wherein: The highlighting of the target person in the display picture at the kth moment in a first preset manner includes: The target person is used as the display content of the display screen at the kth moment.

9. The method for determining a target person according to claim 8, wherein: When the target person is different from the new target person, highlighting the new target person in the display picture at the k+Lth time according to a second preset manner includes: In a case where the target person is different from the new target person, the current display content of the display screen is switched from the target person to the new target person.

10. The method for determining a target person according to claim 9, wherein: When the target person is different from the new target person, switching the current display content of the display screen from the target person to the new target person includes: When the target person is different from the new target person, the size and position of the display content in the display screen are uniformly changed based on the size information and position information of the target person and the size information and position information of the new target person, until the display content in the display screen at the k+Lth moment is the new target person.

11. A device for determining a target person, characterized in that: include: A first determining unit is used to determine n persons to be detected in the display picture at the kth moment, where k and n are both positive integers; An acquisition unit, used for acquiring speech feature data related to the n persons to be tested from the display screen; The second determining unit is used to determine the target person in the display image at the kth moment from the n persons to be tested based on the speech feature data.

12. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

13. A computer program product, characterized in that The invention comprises a computer program, and when the computer program is executed, the method for determining a target person according to any one of claims 1 to 10 is executed.

Images