Crowd type identification method and device
By combining biological data and historical usage data, terminal devices can more accurately identify the user's population type, solving the problem of low recognition accuracy in the prior art and improving the user experience.
Patent Information
- Application Number
- CN202310613419.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-11-26
AI Technical Summary
When identifying user groups, existing terminal devices have low accuracy, resulting in poor user experience, especially it is difficult to accurately distinguish between confusing groups such as adolescence and adults and the elderly.
By combining the biological data currently used by the user (such as sound data, breathing data, fingerprint data) and historical usage data, we determine whether the user is a user who has been using terminal devices for a long time, and improve population type identification based on the fusion biological data and historical usage data. Accuracy.
It improves the accuracy of crowd type identification, enhances user experience, and reduces misjudgment, especially when distinguishing crowd types that are prone to confusion.
Smart Images

Figure CN119026103A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal technology, and in particular to a method and device for identifying crowd types. Background Art
[0002] When a user uses a terminal device, the terminal device can identify the user's demographic type, such as whether the user is a minor, so as to provide the user with more reasonable services. For example, when a user opens a game application on a terminal device, the terminal device can identify whether the user is a minor. If the user is a minor, the terminal device can limit the user's use of the game application to a preset time. Currently, terminal devices usually obtain the user's facial features based on the user's image data; and then determine the user's demographic type based on the user's facial features.
[0003] However, the accuracy of identifying the user's crowd type based on the user's facial features is low, resulting in a poor user experience. Summary of the invention
[0004] The present application provides a method and apparatus for identifying crowd types, which can improve the accuracy of a terminal device in identifying the crowd type to which a user belongs, thereby improving the user experience.
[0005] In a first aspect, a method for identifying a type of crowd is provided, comprising: upon obtaining first biometric data of a user when currently using a terminal device, determining whether the user is a first type of user based on the first biometric data, the first biometric data comprising at least one of voice data, breathing data, fingerprint data or face data, the first type of user being a user whose usage time and / or usage frequency of the terminal device exceeds a first preset threshold; upon the user being the first type of user, obtaining historical data of the terminal device, the historical data comprising historical usage data of an application on the terminal device and second biometric data of the user when historically using the terminal device, the second biometric data comprising at least one of voice data, breathing data or face data; and determining whether the type of crowd to which the user belongs is a target crowd type based on the historical data.
[0006] The crowd type identification method of the present application determines whether the user is a long-term user of the terminal device based on the biometric data of the user when currently using the terminal device; when the user is a long-term user of the terminal device, the historical biometric data and historical usage data of the historical application program of the user are obtained; and then the crowd type to which the user belongs is determined to be the target crowd type through the historical biometric data and the historical usage data. In this way, in addition to determining the crowd type to which the user belongs through the biometric data, the terminal device can also combine the historical usage data to further assist in determining whether the user belongs to the crowd type. By combining the historical usage data and the biometric data, the accuracy of the crowd type identification method can be improved, and the user experience can be improved.
[0007] It should be understood that the first type of user may also be referred to as a long-term user or an owner of the device. The first preset threshold may be a preset duration and / or number of times. The user's historical use of the terminal device may refer to the user's use of the terminal device at any time before the current time, or may refer to the user's use of the terminal device at any time before the user's current use of the terminal device. The user's usage time and / or usage times of the terminal device may include the time and / or number of times the terminal device is used this time. The usage time and / or usage times refer to the user's historical usage time and / or usage times of the current terminal device. The target population type may be various population types such as minors, the elderly, teachers, women, etc.
[0008] In certain implementations of the first aspect, determining whether the type of population to which the user belongs is a target population type based on the historical data includes: when the first biometric data includes at least one of sound data, breathing data or face data, determining whether the type of population to which the user belongs is the target population type based on the historical data and the first biometric data.
[0009] It should be understood that the terminal device can determine the age group to which the user belongs through at least one of the user's face data, voice data or face data. In the case where the first biometric data does not include voice data, breathing data and face data, the accuracy of the terminal device determining the age group to which the user belongs through fingerprint data is low. Therefore, in the case where the first biometric data includes at least one of voice data, breathing data or face data, the terminal device determines whether the type of crowd to which the user belongs is the target crowd type based on the first biometric data, the second biometric data and the historical usage data. In this way, the accuracy of the crowd type recognition method can be further improved.
[0010] In certain implementations of the first aspect, determining whether the type of population to which the user belongs is the target population type based on the historical data and the first biometric data includes: fusing the first biometric data and the second biometric data to obtain fused biometric data; determining first age information of the user based on the fused biometric data; and determining whether the type of population to which the user belongs is the target population type based on the first age information and the historical usage data.
[0011] It should be understood that fusing the first biometric data with the second biometric data may also be referred to as merging or concatenating the first biometric data with the second biometric data. The fused biometric data includes at least one of face data, voice data, or breathing data. The fused biometric data includes the biometric data of the user currently using the terminal device and the biometric data of the user's historical use of the terminal device, which helps to improve the accuracy of the user's first age information determined by the terminal device.
[0012] In certain implementations of the first aspect, the target population type is a minor type; the method further includes: determining a second age information of the user based on the historical usage data; the determining whether the population type to which the user belongs is the target population type includes: determining whether the population type to which the user belongs is the minor type based on the first age information and the second age information.
[0013] It should be understood that the living habits and terminal device usage habits of people of different age groups may be different, so that the historical usage data stored in the terminal device is different. In this way, the terminal device can determine the second age information of the user based on the historical usage data. It can be understood that the biometrics of some adolescents are close to those of adults. Therefore, when the terminal device determines whether adolescents and adults with biometrics close to those of minors are of the type of minors based on the fusion of biometrics, the accuracy is low. With the assistance of the historical usage data, the terminal device can further determine the second age information of the user based on the historical usage data. In this way, based on the first age information and the second age information, the accuracy of the terminal device in determining whether the type of people to which the user belongs is the type of minor can be improved. Exemplarily, the first age information indicates that the user is an adult, and the age group to which the user belongs is expected to be 20-25 years old; the historical usage data indicates that the place where the user goes in and out every day on weekdays is a middle school, the time period for using the terminal device is from 20:00 to 21:00, and the application with the most usage is a game application. The second age information determined by the terminal device based on the historical usage data indicates that the user is a middle school student, that is, a minor. Since the user's age range determined by the terminal device based on the fused biometric data is 20-25 years old, and the biometric characteristics of people in the 20-25 age range are similar to those of some adolescents, the accuracy of the first age information determined by the terminal device based on the fused biometric data is low. Combined with historical usage data, the user goes in and out of a middle school every day on weekdays, but an adult aged 20-25 is more likely to be a college student, and the place he goes in and out every day should be a university. Combined with historical usage data such as the user's most frequently used applications are game applications, the terminal device can determine that the user is more likely to be a middle school student. Therefore, the final judgment result of the terminal device is that the user is a minor.
[0014] In certain implementations of the first aspect, determining the first age information of the user based on the fused biometric data includes: when the fused biometric data includes face data and voice data, inputting the fused biometric data into a face-human speech sound model to obtain the first age information, wherein the face-human speech sound model is obtained by weighted fusion of an output result of a face model and an output result of a human speech sound model; or, when the fused biometric data includes face data and breathing data, inputting the fused biometric data into a face-breathing model to obtain the first age information, wherein the face-breathing model is obtained by weighted fusion of an output result of a face model and an output result of a breathing model; or, when the fused biometric data includes face data, voice data and breathing data, inputting the face data and voice data in the fused biometric data into the face-human speech sound model to obtain the first age information.
[0015] It should be understood that the face-human speech sound model can determine the first age information of the user based on the face data and the sound data. The face-human speech sound model has the functions of the face model and the human speech sound model, and the output result of the face model and the output result of the human speech sound model are weighted and fused to obtain the first biological information. The face-respiration model can determine the first age information of the user based on the face data and the respiration data. The face-respiration model has the functions of the face model and the respiration model, and the output result of the face model and the output result of the human speech sound model are weighted and fused to obtain the first biological information. The face model is a model that can output the age information of the user based on the face data. The face model can include a face decision function, and the age information of the user can be determined based on the face data through the face decision function; the human speech sound model is a model that can output the age information of the user based on the sound data; the respiration model is a model that can output the age information of the user based on the respiration data.
[0016] In certain implementations of the first aspect, the facial data is valid facial data that has been screened.
[0017] It should be understood that the user's facial data obtained by the terminal device may contain invalid facial data such as poor clarity and image data that does not contain the user's complete face. Based on the invalid facial data, it is impossible to obtain a facial feature vector or facial feature matrix. Therefore, the terminal device can first screen the facial data, screen the valid facial data, and remove the invalid facial data. This helps to improve the accuracy of the first age information.
[0018] In certain implementations of the first aspect, the method further includes: in the case where the first biometric data is not obtained, obtaining interaction data of the user when currently using the terminal device, the interaction data including at least one of sensor data, touch screen data, or application type data; based on the interaction data, determining whether the type of population to which the user belongs is the target population type.
[0019] It should be understood that if the first biometric data is not obtained, the terminal device cannot determine whether the user is a first type of user based on the first biometric data. Therefore, the terminal device cannot obtain the historical data of the user's historical use of the terminal device. In this case, the terminal device can obtain the interaction data of the user when currently using the terminal device. The interaction data of the user when currently using the terminal device can refer to at least one of the sensor data, touch screen data, or application type data obtained by the terminal device during the user's current use of the terminal device. Sensor data may include data from sensors such as pressure sensors and temperature sensors; touch screen data may include data such as the sliding length and area of the user's finger when sliding on the display screen of the terminal device; application type data may include data such as the application type and / or application name used by the user when using the terminal device this time.
[0020] In certain implementations of the first aspect, the method further includes: when the user is not the first type of user, obtaining interaction data of the user when currently using the terminal device, the interaction data including at least one of sensor data, touch screen data, or application type data; based on the interaction data and / or the first biometric data, determining whether the type of population to which the user belongs is the target population type.
[0021] It should be understood that when the user is not a first type of user, the terminal device cannot obtain historical data of the user's historical use of the terminal device. In this way, the terminal device can obtain the interaction data of the user when currently using the terminal device, and determine whether the type of population the user belongs to is the target population type based on the interaction data and / or the first biometric data.
[0022] In certain implementations of the first aspect, before determining whether the type of population to which the user belongs is the target population type, the method further includes: based on the interaction data, selecting extended interaction data of a first age group from extended interaction data of multiple age groups, the extended interaction data of the multiple age groups being obtained by clustering sample interaction data, the sample interaction data including at least one of sensor data, touch screen data or application type data of terminal devices used by multiple users in a historical time period; fusing the interaction data with the extended interaction data of the first age group to obtain fused interaction data; determining whether the type of population to which the user belongs is the target population type includes: determining whether the user is the target population type based on the fused interaction data.
[0023] It should be understood that the duration of the historical time period can be any duration. The multiple users may include users of different age groups. After clustering at least one of the sensor data, touch screen data, or application type data of the terminal device used by multiple users in the historical time period, extended interaction data of multiple age groups can be obtained. The extended interaction data includes at least one of the sensor data, touch screen data, or application type data. Exemplarily, the sensor data, touch screen data, and application type data of the terminal device used by 1,000 users in the historical time period are clustered to obtain extended interaction data for the age group of 10 to 20 years old, extended interaction data for the age group of 20 to 40 years old, extended interaction data for the age group of 40 to 60 years old, and extended interaction data for the age group of 60 to 70 years old; the extended interaction data of different age groups are different.
[0024] In certain implementations of the first aspect, before selecting the extended interaction data of the first age group from the extended interaction data of multiple age groups, the method further includes: using an age model to extract features from the sample interaction data to obtain feature data; and performing cluster analysis on the feature data to obtain the extended interaction data of the multiple age groups.
[0025] It should be understood that the terminal device can perform feature extraction on the sensor data, touch screen data and application type data in the sample interaction data respectively through the age model. Specifically, the terminal device inputs the sensor data, touch screen data and application type data in the sample interaction data into the age model respectively, and the age model can output feature data of three modalities, namely, feature data of the sensor modality, feature data of the touch screen modality and feature data of the application type modality. The feature data can be a feature vector or a feature matrix. The three modalities can refer to the sensor modality, the touch screen modality and the application type modality respectively. The age model can be a deep learning convolution model. The feature data output by the age model can include different age labels respectively. The age label can refer to information used to indicate any age, or information used to indicate an age group. The terminal device clusters the feature data of the three modalities to obtain extended interaction data of multiple age groups, and the extended interaction data of multiple age groups can also be called codebook features of multiple age groups. Each of the multiple age groups can include different extended interaction data respectively. The extended interaction data of multiple age groups can also be called a feature pattern library.
[0026] In certain implementations of the first aspect, before fusing the interaction data with the extended interaction data of the first age group, the method also includes: performing dimensionality reduction processing on the interaction data to obtain reduced dimensionality interaction data; performing dimensionality reduction processing on the extended interaction data of the first age group to obtain reduced dimensionality extended interaction data; the fusing of the interaction data with the extended interaction data of the first age group includes: fusing the reduced dimensionality interaction data with the reduced dimensionality extended interaction data to obtain the fused interaction data.
[0027] It should be understood that the dimension of the interaction data and the extended interaction data can be reduced by dimensionality reduction processing. Dimensionality reduction processing of the interaction data can refer to dimensionality reduction processing of the data of the three modes in the interaction data respectively, that is, reducing the dimension of the sensor data in the interaction data; reducing the dimension of the touch screen data in the interaction data; reducing the dimension of the application type data in the interaction data.
[0028] In certain implementations of the first aspect, the interaction data includes the reciprocal N pieces of touch screen data before the moment when the interaction data is acquired, where N is a positive integer.
[0029] It should be understood that N can be a preset positive integer.
[0030] In certain implementations of the first aspect, the interaction data includes M pieces of filtered valid touch screen data, where M is a positive integer.
[0031] It should be understood that M is any positive integer less than or equal to N. The M pieces of valid touch screen data may be data obtained by screening the reciprocal N pieces of touch screen data before the moment of acquiring the interaction data.
[0032] In certain implementations of the first aspect, the method further includes: determining the time interval between adjacent touch screen data among the N last touch screen data before the moment of obtaining the interaction data, where N is a positive integer greater than or equal to M; determining a substitution time node according to the time interval; and determining the touch screen data after the substitution time node that is closest to the moment of obtaining the interaction data among the substitution time nodes as the M valid touch screen data.
[0033] It should be understood that the generation time of each touch screen data may refer to the time when the user touches the screen so that the terminal device obtains and stores the touch screen data. Adjacent touch screen data refers to any two adjacent touch screen data after sorting the N touch screen data according to the generation time. Exemplarily, the N touch screen data include a touch screen data generated at 10:00 on January 1, 2023, a touch screen data at 10:05 on January 1, 2023, a touch screen data at 9:00 on January 1, 2023, and a touch screen data at 12:00 on January 1, 2023; after sorting the N touch screen data, in order from early to late: a touch screen data at 9:00 on January 1, 2023, a touch screen data at 10:00 on January 1, 2023, a touch screen data at 10:05 on January 1, 2023, and a touch screen data at 12:00 on January 1, 2023. Touch screen data and a touch screen data at 12:00 on January 1, 2023; the time interval between two adjacent touch screen data refers to the time interval between a touch screen data at 9:00 on January 1, 2023 and a touch screen data at 10:00 on January 1, 2023, the time interval between a touch screen data at 10:00 on January 1, 2023 and a touch screen data at 10:05 on January 1, 2023, and the time interval between a touch screen data at 10:05 on January 1, 2023 and a touch screen data at 12:00 on January 1, 2023.
[0034] In certain implementations of the first aspect, before determining whether the type of population to which the user belongs is the target population type, the method further includes: determining a feedback delay based on the business requirements of the current business of the terminal device; when the feedback delay is greater than or equal to a second preset threshold, obtaining third biometric data of the user using the terminal device within a preset time period, the third biometric data including at least one of sound data, breathing data or face data; determining whether the type of population to which the user belongs is the target population type based on the historical data includes: determining whether the type of population to which the user belongs is the target population type based on the historical data and the third biometric data.
[0035] It should be understood that the current business of the terminal device may refer to the business that triggers the crowd type identification method. For example, when a user is using the payment service of the terminal device, the crowd type identification method is needed to identify whether the user is a minor, and the current business of the terminal device may refer to the payment service. The second preset threshold value may be a preset duration, such as 5 minutes. The preset time period may be a preset duration, such as 3 minutes, 10 minutes, etc. In one example, the second preset threshold value is 6 minutes, and the preset time period is 5 minutes. When the user uses the payment service, it is necessary to determine whether the user is a minor. The feedback delay of the payment service is 50 seconds. The feedback delay of the current service is less than the second preset threshold value. Therefore, the terminal device will not obtain the third biometric data, but will determine whether the crowd type to which the user belongs is a minor type based on historical data.
[0036] In a second aspect, a crowd type identification device is provided, which is used to execute the method in any possible implementation of the first aspect. Specifically, the device includes a module for executing the method in any possible implementation of the first aspect.
[0037] In a third aspect, another crowd type recognition device is provided, including a processor, the processor is coupled to a memory, and can be used to execute instructions in the memory to implement the method in any possible implementation of the first aspect. Optionally, the device also includes a memory. Optionally, the device also includes a communication interface, and the processor is coupled to the communication interface.
[0038] In one implementation, the apparatus is a terminal device. When the apparatus is a terminal device, the communication interface may be a transceiver, or an input / output interface.
[0039] In another implementation, the device is a chip configured in a terminal device. When the device is a chip configured in a terminal device, the communication interface may be an input / output interface.
[0040] In a fourth aspect, a processor is provided, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is used to receive a signal through the input circuit and transmit a signal through the output circuit, so that the processor executes the method in any possible implementation of the first aspect.
[0041] In the specific implementation process, the processor may be a chip, the input circuit may be an input pin, the output circuit may be an output pin, and the processing circuit may be a transistor, a gate circuit, a trigger, and various logic circuits. The input signal received by the input circuit may be received and input by, for example, but not limited to, a receiver, and the signal output by the output circuit may be, for example, but not limited to, output to a transmitter and transmitted by the transmitter, and the input circuit and the output circuit may be the same circuit, which is used as an input circuit and an output circuit at different times. The embodiments of the present application do not limit the specific implementation methods of the processor and various circuits.
[0042] In a fifth aspect, a processing device is provided, comprising a processor and a memory. The processor is used to read instructions stored in the memory, and can receive signals through a receiver and transmit signals through a transmitter to execute the method in any possible implementation of the first aspect.
[0043] Optionally, the number of the processors is one or more, and the number of the memories is one or more.
[0044] Optionally, the memory may be integrated with the processor, or the memory may be provided separately from the processor.
[0045] In the specific implementation process, the memory can be a non-transitory memory, such as a read-only memory (ROM), which can be integrated with the processor on the same chip or can be set on different chips respectively. This application does not limit the type of memory and the setting method of the memory and the processor.
[0046] It should be understood that the relevant data interaction process, such as sending indication information, can be a process of outputting indication information from a processor, and receiving capability information can be a process of receiving input capability information from a processor. Specifically, the processed output data can be output to a transmitter, and the input data received by the processor can come from a receiver. Among them, the transmitter and the receiver can be collectively referred to as a transceiver.
[0047] The processing device in the fifth aspect mentioned above can be a chip. The processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor, which is implemented by reading the software code stored in the memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.
[0048] In a sixth aspect, a computer program product is provided, the computer program product comprising: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute a method in any possible implementation of the first aspect.
[0049] In a seventh aspect, a computer-readable storage medium is provided, which stores a computer program (also referred to as code, or instructions) which, when executed on a computer, enables the computer to execute a method in any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;
[0051] Figure 2 A software structure block diagram of a terminal device according to an embodiment of the present application;
[0052] Figure 3 A schematic diagram of an application scenario is provided for an embodiment of the present application;
[0053] Figure 4 A flow chart of a crowd type identification method provided in an embodiment of the present application;
[0054] Figure 5 A schematic diagram of a process of a method for identifying a crowd type when a user is a first type of user provided in an embodiment of the present application;
[0055] Figure 6 A process diagram of a method for identifying a crowd type when a user provided in an embodiment of the present application is a first type user or not a first type user;
[0056] Figure 7 A schematic diagram of a process of a method for identifying a population type when a user provided in an embodiment of the present application is not a first type of user or the first biometric data is not obtained;
[0057] Figure 8 A schematic diagram of another crowd type identification method provided in an embodiment of the present application;
[0058] Fig. 9 A schematic diagram of a terminal device display interface provided in an embodiment of the present application;
[0059] Fig.10 A schematic block diagram of a crowd type identification device provided in an embodiment of the present application;
[0060] Fig.11 A schematic block diagram of another crowd type identification device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0061] The technical solution in this application will be described below in conjunction with the accompanying drawings.
[0062] In the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially identical functions and effects. For example, the first value and the second value are only used to distinguish between different values, and do not limit their order. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.
[0063] It should be noted that in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.
[0064] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.
[0065] The terminal device provided in the embodiment of the present application is a terminal device with a touch screen, which can be a mobile phone, a tablet computer (pad), a desktop computer, a laptop computer, etc. The embodiment of the present application does not limit the specific technology and specific device form adopted by the terminal device.
[0066] In order to better understand the terminal device in the embodiment of the present application, Figure 1 The hardware structure of the terminal device in the embodiment of the present application is described in detail.
[0067] Figure 1 This is a schematic diagram of the structure of the terminal device 100 provided in the embodiment of the present application. Figure 1As shown, the terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0068] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0069] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0070] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.
[0071] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0072] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0073] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal device 100.
[0074] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.
[0075] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
[0076] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.
[0077] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.
[0078] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.
[0079] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminal devices, such as AR devices, etc.
[0080] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0081] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the terminal device through the power management module 141.
[0082] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.
[0083] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
[0084] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0085] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.
[0086] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.
[0087] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0088] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).
[0089] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.
[0090] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0091] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0092] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.
[0093] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0094] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0095] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0096] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0097] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0098] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.
[0099] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.
[0100] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.
[0101] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.
[0102] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.
[0103] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.
[0104] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
[0105] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 100 detects the touch operation intensity according to the pressure sensor 180A. The terminal device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.
[0106] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.
[0107] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
[0108] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.
[0109] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in various directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device and applied to applications such as horizontal and vertical screen switching and pedometers.
[0110] The distance sensor 180F is used to measure the distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.
[0111] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.
[0112] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touch.
[0113] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.
[0114] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to avoid abnormal shutdown of the terminal device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
[0115] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 100, which is different from the position of the display screen 194.
[0116] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.
[0117] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.
[0118] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.
[0119] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.
[0120] The SIM card interface 195 is used to connect the SIM card. The SIM card can be connected to and separated from the terminal device 100 by inserting the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to realize functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100. The software system of the terminal device 100 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system with a layered architecture as an example to illustrate the software structure of the terminal device 100.
[0121] Figure 2 This is a software structure block diagram of the terminal device 100 according to an embodiment of the present application.
[0122] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.
[0123] The application layer can include a series of application packages. Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.
[0124] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.
[0125] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.
[0126] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0127] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.
[0128] The phone manager is used to provide communication functions of the terminal device 100, such as management of call status (including connection, disconnection, etc.).
[0129] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
[0130] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, the terminal device vibrates, the indicator light flashes, etc.
[0131] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.
[0132] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.
[0133] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0134] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.
[0135] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.
[0136] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0137] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis and layer processing.
[0138] A 2D graphics engine is a drawing engine for 2D drawings.
[0139] The kernel layer is a layer between hardware and software. The kernel layer is used to drive the hardware to make the hardware work. The kernel layer at least includes display driver, screen driver, graphics processing unit (GPU) driver, camera, and sensor driver, etc., which are not limited in the embodiments of the present application. For example, the screen driver can drive the screen to turn on or off.
[0140] When a user uses a terminal device, the terminal device can identify the user's population type, so as to provide the user with more reasonable services. The population type can be minors, adults, the elderly, middle school students, college students, or a certain profession, etc. A certain profession can be a teacher, a chef, etc. For example, Figure 3 A schematic diagram of an application scenario 300 is provided for an embodiment of the present application. Figure 3 As shown in the figure, when the user plays games on the mobile phone, Figure 3 As shown in the interface (a) in the figure, the user recharges the game. When choosing to recharge 200 yuan, the display interface of the mobile phone jumps to Figure 3 Interface (b) is displayed, where the user can select a payment method to make the payment. However, due to the frequent occurrence of minors recharging games and giving live broadcast rewards, before the user makes a payment, the mobile phone can identify the user's demographic type, such as whether the user is a minor. When the mobile phone obtains the user's confidence level as a minor, for example, if the user's confidence level as a minor is 90%, the mobile phone determines that the user is a minor. At this time, if the user Figure 3 If the password in the interface (b) shown is entered correctly, the mobile phone can further display the parent account and password verification interface. If the verification is passed, the payment is successful; otherwise, the payment fails.
[0141] At present, the terminal device usually obtains the user's facial features based on the user's image data; then inputs the user's facial features into the face model to obtain the user's crowd type. However, the accuracy of such crowd type recognition methods is low, resulting in a poor user experience. For example, the terminal device determines whether the user is a minor based on the user's facial features. Assuming that the user is a minor in adolescence, since the facial features of some minors in adolescence are similar to those of adults, the terminal device may determine that the user is an adult. Similarly, when the terminal device determines whether the user is an elderly person, since the facial features of some middle-aged people are similar to those of the elderly, the terminal device may determine that the middle-aged person is an elderly person. Therefore, for easily confused crowd types such as adolescents and middle-aged people, the terminal device may make wrong judgments.
[0142] In order to solve the above problems, the present application provides a crowd type identification method. When obtaining the user's biometric data, such as face data, voice data, breathing data or fingerprint data when the user uses a terminal device, the terminal device can determine whether the user is a long-term user of the terminal device based on the user's biometric data; if the user is a long-term user of the terminal device, the terminal device will obtain the user's historical biometric data when the terminal device was used historically, as well as the historical usage data of the application stored in the terminal device; the terminal device determines the user's crowd type based on the historical usage data, historical biometric data and biometric data. In this way, on the one hand, the terminal device determines the user's age group based on the historical biometric data and biometric data; on the other hand, the terminal device can determine the user's habit information based on the historical usage data, such as living habit information, work habit information, etc., and then, the terminal device can further determine the user's crowd type based on the user's habit information and the user's age group determined based on the biometric data and historical biometric data, thereby improving the accuracy of the crowd type identification method and improving the user experience.
[0143] Combine the following Figures 4 to 9 The crowd type identification method of the present application is described in detail. The execution subject of the embodiment shown in the present application can be a terminal device, and the hardware structure of the terminal device can be as follows Figure 1 As shown, the software structure can be as follows Figure 2 Below, taking the terminal device as an example, the crowd type identification method of the embodiment of the present application is described in detail.
[0144] It should be understood that the terminal device can be the terminal device itself, or it can be a chip, chip system or processor that supports the terminal device to implement a crowd type recognition method, or it can be a logic module or software that can implement all or part of the terminal device functions. This application does not make specific limitations on this.
[0145] Figure 4 4 is a flow chart of a crowd type identification method 400 provided in an embodiment of the present application. Figure 4 As shown, the method 400 includes the following steps:
[0146] S401. When first biometric data of a user currently using a terminal device is obtained, determine whether the user is a first type of user based on the first biometric data, where the first biometric data includes at least one of sound data, breathing data, fingerprint data or face data. The first type of user is a user whose usage time and / or usage frequency of the terminal device exceeds a first preset threshold.
[0147] It should be understood that the user's current use of the terminal device may refer to the process from the user's start of using the terminal device to the current moment. For example, the current moment is 10:00 on January 15, 2023, and the user unlocks the terminal device and starts using the terminal device at 9:00 on January 15, 2023, then the user's current use of the terminal device may be from 9:00 on January 15, 2023 to 10:00 on January 15, 2023. Correspondingly, the user's historical use of the terminal device may refer to the process of the user using the terminal device before the current use of the terminal device. For example, the current moment is 10:00 on January 15, 2023, and the user unlocks the terminal device and starts using the terminal device at 9:00 on January 15, 2023, then the user's historical use of the terminal device may be any time period before 9:00 on January 15, 2023.
[0148] In a possible implementation, the first biometric data may be obtained after the user unlocks the terminal device. In this case, when the user is currently using the terminal device, the time period from when the user's first biometric data is obtained to when the terminal device is last unlocked may be referred to. Exemplarily, the user unlocks the mobile phone at 11:00 on January 1, 2023, and triggers method 400 at 12:00 on January 1, 2023, that is, the mobile phone obtains the first biometric data at 12:00 on January 1, 2023, then the first biometric data is the biometric data collected by the mobile phone from 11:00 on January 1, 2023 to 12:00 on January 1, 2023. The first biometric data may include the biometric data collected by the terminal device when the user unlocks the terminal device.
[0149] It can be understood that after the user unlocks the terminal device, the terminal device can trigger method 400 in different situations to obtain the first biometric data. In one case, after the user unlocks the terminal device, when using certain services in the terminal device, the terminal device can trigger method 400, that is, obtain the first biometric data. For example, when the user uses the terminal device for payment services, the terminal device collects the first biometric data. In another case, the terminal device periodically triggers method 400 to obtain the first biometric data. For example, the terminal device triggers method 400 every 48 hours to obtain the first biometric data.
[0150] In another possible implementation, the first biometric data can be obtained when the user unlocks the terminal device, and based on the first biometric data, it is determined whether the user is a first type of user. Exemplarily, the user unlocks the phone through face recognition, so that the phone can obtain the user's face image, and obtain face data based on the face image, and then further determine whether the user is a first type of user based on the face data.
[0151] It should be understood that unlocking a terminal device can also be called authentication. Users can unlock the terminal device by face unlocking, password unlocking, fingerprint unlocking, etc. This application does not make specific limitations on this.
[0152] The sound data may be the audio data obtained by the terminal device during the user's speaking, voice call, singing, video call, using a voice assistant, etc. The face data may be the image data obtained by the terminal device during the user's unlocking of the terminal device, video call, etc. The breathing data may be the audio data obtained by the terminal device during any process that can obtain the user's audio data, such as a voice call or video call. The fingerprint data may be the sensor data obtained by the terminal device during the user's unlocking of the terminal device by fingerprint, etc.
[0153] The first type of user may also be referred to as a long-term user or the owner of the device. The first preset threshold may be a preset duration and / or number of times. In the case where the first type of user is a user whose usage time of the terminal device exceeds the first preset threshold, the first preset threshold is a preset duration, such as 20 hours. When the terminal device determines that the total duration of the user's historical use of the terminal device exceeds 20 hours, the user is a first type of user. In the case where the first type of user is a user whose number of uses of the terminal device exceeds the first preset threshold, the first preset threshold is a preset number of times, such as 10 times. When the terminal device determines that the number of times the user's historical use of the terminal device exceeds 10 times, the user is a first type of user. In the case where the first type of user is a user whose usage time and number of uses of the terminal device exceed the first preset threshold, the first preset threshold includes a preset duration and a preset number of times. For example, the first preset threshold includes 20 hours and 10 times. When the total duration of the user's historical use of the terminal device exceeds 20 hours and the total number of historical uses of the terminal device exceeds 10 times, the user is a first type of user.
[0154] It can be understood that the user's historical use of the terminal device may refer to the user's use of the terminal device at any time before the current time, or may refer to the user's use of the terminal device at any time before the user's current use of the terminal device. The user's use time and / or use frequency of the terminal device may include the time and / or number of times the terminal device is used this time. The use time and / or use frequency refers to the user's historical use time and / or use frequency of the current terminal device.
[0155] In a possible implementation, the first type of user is a user whose usage time and / or usage frequency of the terminal device exceeds a first preset threshold value within a first preset historical time period, and the first preset historical time period is a time period before the current moment. Exemplarily, the first preset historical time period is a week before the current moment, and if the user's usage time and / or usage frequency of the terminal device within the previous week exceeds the first preset threshold value, the user is a long-term user of the terminal device, that is, the user is a first type of user.
[0156] S402. In case the user is a first type user, historical data of the terminal device is obtained, where the historical data includes historical usage data of applications on the terminal device and second biometric data of the user when historically using the terminal device, where the second biometric data includes at least one of voice data, breathing data or face data.
[0157] It should be understood that the historical usage data of an application on a terminal device may refer to any historical usage data of an application stored on the terminal device, such as the name of the application, the time when the user uses the application, the business performed by the user through the application, and other data. Optionally, the historical usage data is the historical usage data of a preset type of application on the terminal device. Exemplarily, the preset type of application may include a navigation application and a takeaway application, and correspondingly, the historical usage data may include geographic location data and takeaway food type data. Optionally, the historical usage data is the historical usage data of the application within a second preset historical time period on the terminal device. Exemplarily, the second preset historical time period is one month before the current moment, and the historical usage data is the usage data of the application within the previous month stored in the terminal device.
[0158] When it is determined that the user currently using the terminal device is a first type of user, the terminal device may obtain the second biometric data of the user cached by the terminal device when the user used the terminal device in the past. For example, if the user made a voice call when using the terminal device in the past, the terminal device may cache the voice data and breathing data of the user; if the user unlocked the terminal device through face recognition when using the terminal device in the past, the terminal device may cache the face data of the user.
[0159] S403: Based on historical data, determine whether the population type to which the user belongs is the target population type.
[0160] It should be understood that the target population type can be various population types such as minors, the elderly, teachers, women, etc. The historical data includes the second biometric data and the historical usage data. The terminal device can obtain the biometric data through the second biometric data. The biometric data can be data such as a biometric vector or a biometric matrix. The terminal device can obtain the user's historical habit feature data through the historical usage data. The historical habit feature data can also be data such as a habit feature vector or a habit feature matrix. Then, the terminal device determines whether the population type to which the user belongs is the target population type based on the biometric data and the historical habit feature data.
[0161] The crowd type identification method of the present application determines whether the user is a long-term user of the terminal device based on the biometric data of the user when currently using the terminal device; when the user is a long-term user of the terminal device, the historical biometric data and historical usage data of the historical application program of the user are obtained; and then the crowd type to which the user belongs is determined to be the target crowd type through the historical biometric data and the historical usage data. In this way, in addition to determining the crowd type to which the user belongs through the biometric data, the terminal device can also combine the historical usage data to further assist in determining whether the user belongs to the crowd type. By combining the historical usage data and the biometric data, the accuracy of the crowd type identification method can be improved, and the user experience can be improved.
[0162] As an optional embodiment, the above S403 can be implemented in the following manner: when the first biometric data includes at least one of sound data, breathing data or face data, determine whether the type of population the user belongs to is the target population type based on historical data and the first biometric data.
[0163] It should be understood that the terminal device can determine the age group to which the user belongs through at least one of the user's face data, voice data or face data. In the case where the first biometric data does not include voice data, breathing data and face data, the accuracy of the terminal device determining the age group to which the user belongs through fingerprint data is low. Therefore, in the case where the first biometric data includes at least one of voice data, breathing data or face data, the terminal device determines whether the type of crowd to which the user belongs is the target crowd type based on the first biometric data, the second biometric data and the historical usage data. In this way, the accuracy of the crowd type recognition method can be further improved.
[0164] As an optional embodiment, determining whether the type of population to which the user belongs is the target population type based on historical data and the first biometric data can be achieved in the following manner: fusing the first biometric data and the second biometric data to obtain fused biometric data; determining the user's first age information based on the fused biometric data; and determining whether the type of population to which the user belongs is the target population type based on the first age information and historical usage data.
[0165] It should be understood that fusing the first biometric data with the second biometric data may also be referred to as merging or concatenating the first biometric data with the second biometric data. The fused biometric data includes at least one of face data, voice data, or breathing data. The fused biometric data includes the biometric data of the user currently using the terminal device and the biometric data of the user's historical use of the terminal device, which helps to improve the accuracy of the user's first age information determined by the terminal device.
[0166] The first age information may include the age group information of the user currently using the terminal device, and may also include the probability information that the user belongs to the age group. For example, if the first age information includes 30-40 and 60%, the probability that the user belongs to 30 to 40 years old is 60%. The first age information may also include multiple age group information and the probability that the user belongs to each of the multiple age groups. For example, if the first age information includes: 30-40 and 60%, 20-30 and 40%, the probability that the user belongs to 30 to 40 years old is 60%, and the probability that the user belongs to 20 to 30 years old is 40%. The first age information may also include other information that can be used to indicate the age information of the user, for example, the first age information may also be a feature vector or a feature matrix.
[0167] The historical usage data may include the user's living habit information, such as the type of places where activities are often conducted, the type of food that is most frequently purchased, the webpages that are viewed the most, the type of books that are read for the longest time, etc.; the historical usage data may also include the user's habit information for using the terminal device, such as the type of application that is used the most, the time period during which the terminal device is most frequently used, etc. The terminal device can more accurately determine whether the type of population to which the user belongs belongs to the target population type by combining the historical usage data and the fused biometric data. Exemplarily, the terminal device determines whether the user currently using the terminal device is a teacher through method 400. On the one hand, the terminal device can determine the age group to which the user belongs by fused biometric data. For example, the terminal device determines that the age group to which the user belongs is most likely 30 to 40 years old; on the other hand, the terminal device can determine through the historical usage data that the user most frequently visits is a middle school. In this way, the terminal device can determine that the user is a teacher based on the historical usage data and the fused biometric data.
[0168] Optionally, determining whether the type of population to which the user belongs is the target population type based on the first age information and the historical usage data includes: determining the user's habit information based on the historical usage data; and determining whether the type of population to which the user belongs is the target population type based on the first age information and the habit information.
[0169] It should be understood that the historical usage data may be the usage data of the application program cached in the terminal device during the historical time period, and the habit information may be determined based on the historical usage data. The habit information may include the user's living habit information, such as the types of places where the user frequently moves, the types of food that the user most frequently buys, the types of books that the user spends the longest time reading, etc. The habit information may also include the user's habit information for using the terminal device, such as the types of application programs that are used the most, the time period when the terminal device is used the most, etc.
[0170] In one possible implementation, the target population type is a minor type; method 400 also includes: determining the user's second age information based on the habit information; determining whether the population type to which the user belongs is the target population type based on the first age information and the historical usage data, including: determining whether the population type to which the user belongs is a minor type based on the first age information and the second age information.
[0171] It should be understood that the living habits and terminal device usage habits of people of different age groups may be different, so that the historical usage data are different. In this way, the terminal device can determine the second age information of the user based on the historical usage data. It can be understood that the biometrics of some adolescents are close to those of adults. Therefore, when the terminal device determines whether the type of population to which adolescents and adults with biometrics close to those of minors belong is a minor type based on the fusion biometrics, the accuracy is low. With the assistance of the historical usage data, the terminal device can further determine the second age information of the user based on the historical usage data. In this way, based on the first age information and the second age information, the accuracy of the terminal device in determining whether the type of population to which the user belongs is a minor type can be improved. Exemplarily, the first age information indicates that the user is an adult, and the age group to which the user belongs is expected to be 20-25 years old; the historical usage data indicates that the place where the user goes in and out every day on weekdays is a middle school, the time period for using the terminal device is from 20:00 to 21:00, and the application with the most usage is a game application. The second age information determined by the terminal device based on the historical usage data indicates that the user is a middle school student, that is, a minor. Since the user's age range determined by the terminal device based on the fused biometric data is 20-25 years old, and the biometric characteristics of people in the 20-25 age range are similar to those of some adolescents, the accuracy of the first age information determined by the terminal device based on the fused biometric data is low. Combined with historical usage data, the user goes in and out of a middle school every day on weekdays, but an adult aged 20-25 is more likely to be a college student, and the place he goes in and out every day should be a university; secondly, combined with historical usage data such as the user's most frequently used applications are game applications, the terminal device can determine that the user is more likely to be a middle school student. Therefore, the final judgment result of the terminal device is that the user is a minor.
[0172] It can be understood that when the target population type is not minors but other age-related population types, such as the elderly, middle-aged people, children and other population types, the terminal device can also determine whether the population type to which the user belongs is the target population type by the same method.
[0173] In a possible implementation, determining the first age information of the user based on the fused biometric data includes:
[0174] In the case where the fused biometric data includes face data and voice data, the fused biometric data is input into a face-human speech voice model to obtain first age information, wherein the face-human speech voice model is obtained by weighted fusion of an output result of the face model and an output result of the human speech voice model; or,
[0175] In the case where the fused biological data includes face data and respiration data, the fused biological data is input into a face-respiration model to obtain first age information, where the face-respiration model is obtained by weighted fusion of an output result of the face model and an output result of the respiration model; or,
[0176] In the case where the fused biometric data includes face data, voice data and breathing data, the face data and voice data in the fused biometric data are input into a face-human speech voice model to obtain the first age information.
[0177] It should be understood that the face-human speech sound model can determine the first age information of the user based on the face data and the sound data. The face-human speech sound model has the functions of the face model and the human speech sound model, and the output result of the face model and the output result of the human speech sound model, such as feature data, confidence, etc., are weighted fused to obtain the first biometric information. The face-respiration model can determine the first age information of the user based on the face data and the respiration data. The face-respiration model has the functions of the face model and the respiration model, and the output result of the face model and the output result of the human speech sound model, such as feature data, confidence, etc., are weighted fused to obtain the first biometric information. The face model is a model that can output the age information of the user based on the face data. The face model can include a face decision function, and the face decision function can determine the age information of the user based on the face data, wherein the face decision function can be a combined decision form of the multi-task output of the face model; the human speech sound model is a model that can output the age information of the user based on the sound data; and the respiration model is a model that can output the age information of the user based on the respiration data.
[0178] The above technical solution obtains the face-human speech sound model by fusing the face model and the human speech sound model, and then obtains the first age information based on the face data and the sound data; or obtains the face-respiration model by fusing the face model and the breathing model, and then obtains the first age information based on the face data and the breathing data. In this way, the sound data or the breathing data is used in combination with the face data to determine the user's age information based on biological data of multiple dimensions, which can improve the accuracy of the first age information. Optionally, in the case of both sound data and breathing data, the sound data is preferentially selected, and the face data and the sound data are input into the face-human speech sound model. In this way, the user's first age information can be determined more accurately than through the face data and the breathing data.
[0179] Optionally, when the fused biometric data includes face data and voice data, before the fused biometric data is input into the face-human speech voice model, the face data can be converted into a face feature vector, and the voice data can be converted into a voice feature vector through different convolutional neural networks; or the face data can be converted into a face feature matrix, and the voice data can be converted into a voice feature matrix. The face-human speech voice model can determine the user's first age information based on the input face feature vector and voice feature vector, or the face feature matrix and voice feature matrix.
[0180] Optionally, when the fused biometric data includes face data and respiration data, before the fused biometric data is input into the face-respiration model, the face data may be converted into a face feature vector, and the respiration data may be converted into a respiration feature vector, respectively, through different convolutional neural networks or other types of machine learning models; or the face data may be converted into a face feature matrix, and the respiration data may be converted into a respiration feature matrix. The face-respiration model can determine the user's first age information based on the input face feature vector and respiration feature vector, or the face feature matrix and respiration feature matrix.
[0181] In a possible implementation, when the fused biometric data includes facial data, the terminal device obtains a facial feature vector or a facial feature matrix based on the facial data; the facial feature vector or the facial feature matrix is input into a facial model, a face-breathing model or a face-human speaking sound model to obtain the user's first age information.
[0182] In one possible implementation, when the fused biological data includes voice data, the terminal device obtains a voice feature vector or a voice feature matrix based on the voice data; the voice feature vector or the voice feature matrix is input into a human speech voice model or a face-human speech voice model to obtain the user's first age information.
[0183] In a possible implementation, when the fused biological data includes breathing data, the terminal device obtains a breathing feature vector or a breathing feature matrix based on the breathing data; and inputs the breathing feature vector or the breathing feature matrix into a human speech sound model or a face-breathing model to obtain the user's first age information.
[0184] In a possible implementation, when the fused biological data includes sound data and breathing data, the terminal device obtains a sound feature vector or a sound feature matrix based on the sound data; the sound feature vector or the sound feature matrix is input into a human speech sound model or a face-human speech sound model to obtain the user's first age information.
[0185] In a possible implementation, the facial data in the fused biometric data is screened valid facial data.
[0186] It should be understood that the user's facial data obtained by the terminal device may include invalid facial data such as poor clarity and image data that does not contain the user's complete face. The terminal device may not be able to obtain a facial feature vector or facial feature matrix based on the invalid facial data. Therefore, the terminal device can first filter the facial data, remove the invalid facial data, and filter out the valid facial data. This helps to improve the accuracy of the first age information.
[0187] As an optional embodiment, before S403, method 400 also includes: determining a feedback delay based on the business requirements of the current business of the terminal device; when the feedback delay is greater than or equal to a second preset threshold, obtaining third biometric data of the user using the terminal device within a preset time period, the third biometric data including at least one of sound data, breathing data or face data; based on historical data, determining whether the type of population to which the user belongs is the target population type, including: determining whether the type of population to which the user belongs is the target population type based on the historical data and the third biometric data.
[0188] It should be understood that the current business of the terminal device may refer to the business that triggers the crowd type identification method. For example, when a user uses the payment service of the terminal device, the crowd type identification method is needed to identify whether the user is a minor, and the current business of the terminal device may refer to the payment service. The second preset threshold value may be a preset duration, such as 5s. The preset time period may be a preset duration, such as 3s, 10s, etc. In one example, the second preset threshold value is 6s, and the preset time period is 5s. When the user uses the payment service, it is necessary to determine whether the user is a minor. The feedback delay of the payment service is 1s, and the feedback delay of the current service is less than the second preset threshold value. Therefore, the terminal device will not obtain the third biometric data, but will determine whether the crowd type to which the user belongs is a minor type based on historical data. In another example, the second preset threshold is 6s, the preset time period is 5s, and the user opens the first game application. The terminal device will determine whether the user is a minor. If the user is a minor, when the user uses the first game application for 2 hours, the offline notification message is displayed. The offline notification message is used to remind the user that the currently logged-in account will automatically go offline and exit the game interface, thereby limiting the user's gaming time. For the game service, the feedback delay is 30s, so the feedback delay of the current service is greater than the second preset threshold, so the terminal device will continue to obtain the user's third biometric data within 5s. And based on the historical data and the third biometric data, it is determined whether the user belongs to a minor type.
[0189] Combine the following Figure 5 , taking the target population type as minors and the user as the first type of user as an example, the population type identification method is further explained.
[0190] Figure 5 The process diagram of a method 500 for identifying a crowd type when a user is a first type of user provided in an embodiment of the present application. Figure 5As shown, the face data may include at least one of the face data in the first biometric data, the face data in the second biometric data, or the face data in the third biometric data; the voice data may include at least one of the voice data in the first biometric data, the voice data in the second biometric data, or the voice data in the third biometric data; the breathing data may include at least one of the breathing data in the first biometric data, the breathing data in the second biometric data, or the breathing data in the third biometric data. After the terminal device inputs the face data into the face-image model, the face-image model can output a face feature vector; after the terminal device inputs the voice data or the breathing data into the audio model, the audio model can output a voice feature vector or a breathing feature vector. Specifically, when the voice data is input into the audio model, the audio model will output a voice feature vector, and when the breathing data is input into the audio model, the audio model will output a breathing feature vector. In the case where the voice data and the breathing data exist, the terminal device inputs the voice data into the audio model. Then, the terminal device inputs the sound feature vector or the breathing feature vector, and the face feature vector into the biometric fusion model, which can output the fused biometric features. The terminal device inputs the fused biometric features into the face-human speech sound model or the face-breathing model, and the face-human speech sound model or the face-breathing model can output the first age information. The terminal device inputs the historical usage data and the first age information into the biometric-user portrait fusion model, and the biometric-user portrait fusion model can output whether the user is a minor type or not. Optionally, before the historical usage data is input into the biometric-user portrait fusion model, the feature vector of the historical usage data is first obtained through a deep learning convolution model; the feature vector of the historical usage data and the first age information are input into the biometric-user portrait fusion model, and the output is whether the user is a minor type or not.
[0191] It should be understood that the face-image model and the audio model can be models such as deep learning convolution models, and this application does not specifically limit the model types of the face-image model and the audio model. The biometric fusion model can fuse the face feature vector and the sound feature vector, or merge the face feature vector and the breathing feature vector. The biometric-user portrait fusion model can determine the user's second age information based on historical usage data, and determine whether the user is a minor type based on the first age information and the second age information.
[0192] As an optional embodiment, method 400 also includes: when the first biometric data is not obtained, obtaining interaction data of the user when currently using the terminal device, the interaction data includes at least one of sensor data, touch screen data, or application type data; based on the interaction data, determining whether the population type to which the user belongs is the target population type.
[0193] It should be understood that if the first biometric data is not obtained, it is impossible to determine whether the user is a first type of user based on the first biometric data. Therefore, the terminal device cannot obtain the historical data of the user's historical use of the terminal device. In this case, the terminal device can obtain the interaction data of the user when currently using the terminal device. The interaction data of the user when currently using the terminal device can refer to at least one of the sensor data, touch screen data or application type data obtained by the terminal device during the user's use of the terminal device this time. Sensor data may include data from sensors such as pressure sensors and temperature sensors; touch screen data may include data such as the sliding length and area when the user's finger slides on the display screen of the terminal device; application type data may include data such as the application type and / or application name used by the user when using the terminal device this time.
[0194] As an optional embodiment, method 400 also includes: when the user is not a first type of user, obtaining interaction data of the user when currently using the terminal device, the interaction data including at least one of sensor data, touch screen data or application type data; based on the interaction data and / or the first biometric data, determining whether the type of population to which the user belongs is the target population type.
[0195] It should be understood that when the user is not a first type of user, the terminal device cannot obtain historical data of the user's historical use of the terminal device. In this way, the terminal device can obtain the interaction data of the user when currently using the terminal device, and determine whether the type of population the user belongs to is the target population type based on the interaction data and / or the first biometric data.
[0196] In one possible implementation, when the first biometric data includes at least one of facial data, voice data or respiratory data, whether the type of population the user belongs to is the target population type is determined based on the first biometric data; when the first biometric data does not include facial data, voice data and respiratory data, whether the type of population the user belongs to is the target population type is determined based on the interaction data.
[0197] It should be understood that, compared with the interaction data, the terminal device may have a higher accuracy in determining whether the type of crowd to which the user belongs is the target type of crowd based on at least one of the face data, voice data, or breathing data in the first biological data. Therefore, in the case where the first biological data includes at least one of the face data, voice data, or breathing data, it is determined based on the first biological data whether the type of crowd to which the user belongs is the target type of crowd, which can improve the accuracy of the crowd type recognition method.
[0198] As an optional embodiment, when the first biometric data is not obtained and / or the user is not a first type of user, before determining whether the population type to which the user belongs is the target population type, method 400 also includes: based on the interaction data, selecting extended interaction data of the first age group from extended interaction data of multiple age groups, the extended interaction data of the multiple age groups are obtained by clustering sample interaction data, the sample interaction data including at least one of sensor data, touch screen data or application type data of terminal devices used by multiple users in a historical time period; fusing the interaction data with the extended interaction data of the first age group to obtain fused interaction data; determining whether the population type to which the user belongs is the target population type, including: determining whether the user is the target population type based on the fused interaction data.
[0199] It should be understood that the terminal devices used by multiple users in the historical time period may be different, for example, they may be tablet computers, smart watches, laptop computers, mobile phones, etc. The duration of the historical time period may be of any duration. Multiple users may include users of different age groups. After clustering at least one of the sensor data, touch screen data, or application type data of the terminal devices used by multiple users in the historical time period, extended interaction data of multiple age groups can be obtained. The extended interaction data includes at least one of the sensor data, touch screen data, or application type data. Exemplarily, the sensor data, touch screen data, and application type data of the terminal devices used by 1,000 users in the historical time period are clustered to obtain extended interaction data of the age group of 10 to 20 years old, extended interaction data of the age group of 20 to 40 years old, extended interaction data of the age group of 40 to 60 years old, and extended interaction data of the age group of 60 to 70 years old; the extended interaction data of different age groups are different.
[0200] In a possible implementation, the first age group is determined by a classification algorithm based on the interaction data and the extended interaction data of multiple age groups. It should be understood that through the classification algorithm, it can be determined that the user's age may belong to the first age group, so the extended interaction data and the interaction of the first age group are fused to obtain fused interaction data. The classification algorithm may be a K-nearest neighbor (KNN) classification algorithm, etc. Since the interaction data is the data obtained by the terminal device when the user is currently using the terminal device, the terminal device may obtain less interaction data in a shorter period of time. By fusing the extended interaction data of the first age group with the interaction data, the user's interaction data can be expanded. In this way, by fusing the interaction data, it is possible to more accurately determine whether the user is the target population type.
[0201] In a possible implementation, before selecting extended interaction data of a first age group from extended interaction data of multiple age groups, method 400 further includes: using an age model to perform feature extraction on sample interaction data to obtain feature data; and performing cluster analysis on the feature data to obtain extended interaction data of multiple age groups.
[0202] It should be understood that the terminal device can perform feature extraction on the sensor data, touch screen data and application type data in the sample interaction data respectively through the age model. Specifically, the terminal device inputs the sensor data, touch screen data and application type data in the sample interaction data into the age model respectively, and the age model can output feature data of three modalities, namely, feature data of the sensor modality, feature data of the touch screen modality and feature data of the application type modality. The feature data can be a feature vector or a feature matrix. The three modalities can refer to the sensor modality, the touch screen modality and the application type modality respectively. The age model can be a deep learning convolution model. The feature data output by the age model can include different age labels respectively. The age label can refer to information used to indicate any age, or information used to indicate an age group. The terminal device clusters the feature data of the three modalities to obtain extended interaction data of multiple age groups, and the extended interaction data of multiple age groups can also be called codebook features of multiple age groups. Each of the multiple age groups can include different extended interaction data respectively. The extended interaction data of multiple age groups can also be called a feature pattern library. Cluster analysis can be performed through different clustering methods, such as unsupervised clustering methods. The unsupervised clustering methods may be density-based spatial clustering of applications with noise (DBSCAN), K-means clustering method, etc. K refers to the ability to cluster sample interaction data into K clusters through Kmeans.
[0203] Fusing the interaction data with the extended interaction data of the first age group may refer to fusing the feature data of the interaction data with the extended interaction data of the first age group. The feature data of the interaction data may also be obtained through an age model. Specifically, fusing the interaction data with the extended interaction data of the first age group may refer to tiling and splicing the feature data of the sensor modality in the interaction data with the feature data of the sensor modality in the extended interaction data of the first age group; tiling and splicing the feature data of the touch screen modality in the interaction data with the feature data of the touch screen modality in the extended interaction data of the first age group; tiling and splicing the feature data of the application type modality in the interaction data with the feature data of the application type modality in the extended interaction data of the first age group.
[0204] Combine the following Figure 6 , the method for identifying the population type when the user is a first type user or not a first type user is further explained.
[0205] Figure 6 A process diagram of a crowd type identification method 600 when a user provided in an embodiment of the present application is a first type user or not a first type user. Figure 6 As shown, when the terminal device obtains the first biometric data, it determines whether the user is a first type of user based on the first biometric data. When the user is a first type of user, feature data of the biometric data, such as a biometric feature vector, can be obtained based on the biometric data of the user; feature data of the historical usage data, such as a historical usage feature vector, can be obtained based on the historical usage data. Then, the terminal device determines whether the type of population to which the user belongs is the target population type based on the biometric feature vector and the historical usage feature vector. It can be understood that when the first biometric data includes at least one of facial data, voice data or respiratory data, the biometric data includes the first biometric data and the second biometric data; when the first biometric data includes at least one of facial data, voice data or respiratory data, and the feedback delay of the current service is greater than or equal to the second preset threshold, the biometric data includes the first biometric data, the second biometric data and the third biometric data; when the first biometric data does not include facial data, voice data and respiratory data, and the feedback delay of the current service is greater than or equal to the second preset threshold, the biometric data includes the second biometric data and the third biometric data; when the first biometric data does not include facial data, voice data and respiratory data, and the feedback delay of the current service is less than the second preset threshold, the biometric data includes the second biometric data.
[0206] like Figure 6 As shown, in the case where the user is not a user of the first type, the terminal device obtains the interaction data of the user currently using the terminal device. The terminal device classifies the interaction data based on the interaction data and the extended interaction data of multiple age groups, and determines the extended interaction data of the first age group; the terminal device fuses the extended interaction data of the first age group with the interaction data to obtain the fused interaction data; finally, the terminal device determines whether the type of population to which the user belongs is the target population type based on the fused interaction data and / or biometric data. It can be understood that in the case where the first biometric data includes at least one of face data, voice data or breathing data, the biometric data includes the first biometric data; in the case where the feedback delay of the current service is greater than or equal to the second preset threshold, the biometric data includes the third biometric data. In the case where the first biometric data does not include face data, voice data and breathing data, and the feedback delay of the current service is less than the second preset threshold, the terminal device determines whether the type of population to which the user belongs is the target population type based on the fused interaction data.
[0207] In a possible implementation, before fusing the interaction data with the extended interaction data of the first age group, method 400 also includes: performing dimensionality reduction processing on the interaction data to obtain reduced dimensionality interaction data; performing dimensionality reduction processing on the extended interaction data of the first age group to obtain reduced dimensionality extended interaction data; fusing the interaction data with the extended interaction data of the first age group, including: fusing the reduced dimensionality interaction data with the reduced dimensionality extended interaction data to obtain fused interaction data.
[0208] It should be understood that the dimension of the interaction data and the extended interaction data can be reduced by dimensionality reduction processing. Performing dimensionality reduction processing on the interaction data can refer to performing dimensionality reduction processing on the data of the three modes in the interaction data respectively, that is, reducing the dimension of the sensor data in the interaction data; reducing the dimension of the touch screen data in the interaction data; reducing the dimension of the application type data in the interaction data. Optionally, before performing dimensionality reduction processing on the interaction data, based on the interaction data, feature data of the interaction data is obtained; and dimensionality reduction processing is performed on the feature data of the interaction data. Specifically, the dimension of the feature data of the sensor modality in the interaction data is reduced; the dimension of the feature data of the touch screen modality in the interaction data is reduced; and the dimension of the feature data of the application type modality in the interaction data is reduced.
[0209] Dimensionality reduction processing can reduce the types of data included in the three modal data. For example, the sensor data includes data of three dimensions: pressure sensor data, temperature sensor data, and gyroscope sensor data. By performing dimensionality reduction processing on the sensor data, the data of the gyroscope sensor data dimension can be reduced to obtain data of two dimensions: sensor data and temperature sensor data. Optionally, dimensionality reduction processing can be performed by methods such as principal component analysis (PCA) or partial least squares discriminant analysis (PLSDA). This application does not specifically limit the method used for dimensionality reduction processing.
[0210] The fusion of the reduced-dimensionality interaction data and the reduced-dimensionality extended interaction data can tile and splice the interaction data of each modality in the reduced-dimensionality interaction data with the data identical to each modality in the extended interaction data. Exemplarily, the reduced-dimensionality interaction data includes feature data of the sensor modality after dimensionality reduction, feature data of the touch screen modality after dimensionality reduction, and feature data of the application type modality after dimensionality reduction; the reduced-dimensionality extended data includes extended feature data of the sensor modality after dimensionality reduction, extended feature data of the touch screen modality after dimensionality reduction, and extended feature data of the application type modality after dimensionality reduction. The terminal device tiles and splices the feature data of the sensor modality after dimensionality reduction with the extended feature data of the sensor modality after dimensionality reduction; tiles and splices the feature data of the touch screen modality after dimensionality reduction with the extended feature data of the touch screen modality after dimensionality reduction; tiles and splices the feature data of the application type modality after dimensionality reduction with the extended feature data of the application type modality after dimensionality reduction.
[0211] In a possible implementation, the feature vector of each modality in the dimension reduction and extension interaction data is classified and encoded, and the orthogonal common feature data of the feature vector of each modality in the dimension reduction and extension interaction data is solved; the dimension reduction interaction data and the dimension reduction and extension interaction data are fused to obtain fused interaction data, including: tiling and splicing the orthogonal common feature data of the feature vector of each modality in the dimension reduction and extension interaction data with the feature vector of each modality in the dimension reduction and extension interaction data, respectively, to obtain fused interaction data. Among them, the orthogonal common feature can refer to the common features of the feature vectors of each modality.
[0212] Exemplarily, the dimension reduction and extension interaction data includes a feature vector of a sensor modality, a feature vector of a touch screen modality, and a feature vector of an application type modality; the terminal device respectively solves the orthogonal common feature data of the feature vector of the sensor modality, the orthogonal common feature data of the feature vector of the touch screen modality, and the orthogonal common feature data of the feature vector of the application type modality in the dimension reduction and extension interaction data; the terminal device flattens and splices the orthogonal common feature data of the feature vector of the sensor modality in the dimension reduction and extension interaction data with the feature vector of the sensor modality in the dimension reduction and extension interaction data; flattens and splices the orthogonal common feature data of the feature vector of the touch screen modality in the dimension reduction and extension interaction data with the feature vector of the touch screen modality in the dimension reduction and extension interaction data; flattens and splices the orthogonal common feature data of the feature vector of the application type modality in the dimension reduction and extension interaction data with the feature vector of the application type modality in the dimension reduction and extension interaction data. Among them, the orthogonal common feature data can be an orthogonal common feature vector.
[0213] As an optional embodiment, the interaction data includes the reciprocal N pieces of touch screen data before the moment of acquiring the interaction data, where N is a positive integer.
[0214] It should be understood that N can be a preset positive integer.
[0215] As an optional embodiment, the interaction data includes M pieces of filtered valid touch screen data, where M is a positive integer.
[0216] It should be understood that M is any positive integer less than or equal to N. The M pieces of valid touch screen data may be data obtained by screening the reciprocal N pieces of touch screen data before the moment of acquiring the interaction data.
[0217] Specifically, the method for screening out valid touch screen data can be described as follows.
[0218] In a possible implementation, method 400 also includes: determining the time interval between adjacent touch screen data in N touch screen data pieces according to the generation time of each touch screen data piece in the countdown N touch screen data pieces before the moment of acquiring the interaction data, where N is a positive integer greater than or equal to M; determining a substitution time node according to the time interval; and determining the touch screen data after the substitution time node that is closest to the moment of acquiring the interaction data among the substitution time nodes as M valid touch screen data pieces.
[0219] It should be understood that the generation time of each touch screen data may refer to the time when the user touches the screen so that the terminal device obtains and stores the touch screen data. Adjacent touch screen data refers to any two adjacent touch screen data after sorting the N touch screen data according to the generation time. Exemplarily, the N touch screen data include a touch screen data generated at 10:00 on January 1, 2023, a touch screen data at 10:05 on January 1, 2023, a touch screen data at 9:00 on January 1, 2023, and a touch screen data at 12:00 on January 1, 2023; after sorting the N touch screen data, in order from early to late: a touch screen data at 9:00 on January 1, 2023, a touch screen data at 10:00 on January 1, 2023, a touch screen data at 10:05 on January 1, 2023, and a touch screen data at 12:00 on January 1, 2023. Touch screen data and a touch screen data at 12:00 on January 1, 2023; the time interval between two adjacent touch screen data refers to the time interval between a touch screen data at 9:00 on January 1, 2023 and a touch screen data at 10:00 on January 1, 2023, the time interval between a touch screen data at 10:00 on January 1, 2023 and a touch screen data at 10:05 on January 1, 2023, and the time interval between a touch screen data at 10:05 on January 1, 2023 and a touch screen data at 12:00 on January 1, 2023. According to the time interval, determining the time node of substitution can specifically be: determining two adjacent touch screen data with a time interval greater than or equal to a preset time length as the time node of substitution. In the case where two adjacent touch screen data are determined as the time node of substitution, the terminal device can determine that the two adjacent touch screen data are generated when different users touch the screen according to the time node of substitution. In the case where there is one substitution time node, the substitution time node is the one closest to the moment of acquiring the interactive data; in the case where there are more than one substitution time node, the generation times of the adjacent touch screen data corresponding to the multiple substitution time nodes are all different. At this time, the terminal device can refer to the generation time of the touch screen data that is closer (or farther) to the moment of acquiring the interactive data among the generation times of the two adjacent touch screen data corresponding to each substitution time node in the multiple substitution time nodes, and determine a substitution time node that is closest to the moment of acquiring the interactive data among the multiple substitution time nodes. The touch screen data whose generation time is closer to the moment of acquiring the interactive data and the touch screen data whose generation time is after the generation time of the touch screen data among the adjacent touch screen data corresponding to the one substitution time node are M valid touch screen data.
[0220] Exemplarily, it is assumed that the time interval between two adjacent touch screen data includes the time interval between a touch screen data at 9:00 on January 1, 2023 and a touch screen data at 10:00 on January 1, 2023, the time interval between a touch screen data at 10:00 on January 1, 2023 and a touch screen data at 10:05 on January 1, 2023, the time interval between a touch screen data at 10:05 on January 1, 2023 and a touch screen data at 10:25 on January 1, 2023, and the time interval between 2 The time interval between a touch screen data at 10:25 on January 1, 2023 and a touch screen data at 10:28 on January 1, 2023, and the preset duration is 10 minutes, then the number of replacement time nodes is two, which are a touch screen data at 9:00 on January 1, 2023 and a touch screen data at 10:00 on January 1, 2023, and a touch screen data at 10:05 on January 1, 2023 and a touch screen data at 10:25 on January 1, 2023; taking the two time nodes as The generation time of the touch screen data whose generation time is closer to the time of acquiring the interaction data among the adjacent touch screen data is used as a reference, that is, the generation time of the touch screen data at 9:00 on January 1, 2023 and the touch screen data at 10:00 on January 1, 2023, and the generation time of the touch screen data at 10:05 on January 1, 2023 and the touch screen data at 10:25 on January 1, 2023. The time is used as a reference; the time nodes closer to the time of obtaining the interaction data are a touch screen data at 10:05 on January 1, 2023 and a touch screen data at 10:25 on January 1, 2023; therefore, a touch screen data at 10:25 on January 1, 2023 and the touch screen data after the generation time of the touch screen data are M valid touch screen data, and the M valid touch screen data are a touch screen data at 10:25 on January 1, 2023 and a touch screen data at 10:28 on January 1, 2023.
[0221] In one possible implementation, a pseudo-labeling method is combined with an adversarial learning method to adaptively migrate the extended interaction data and weak models of multiple age groups based on the device type of the terminal device currently used by the user, so that the extended interaction data of multiple age groups are more suitable for the device to identify the user's population type.
[0222] It should be understood that the adaptive migration process can be a process of changing the sample interaction data according to the device type of the terminal device currently used by the user. Exemplarily, the sample interaction data includes the touch screen data of multiple users using the terminal device in the historical time period; in the sample interaction data, the terminal devices used by multiple users in the historical time period may include tablet computers, smart watches and other devices. Assuming that the terminal device currently used by the user is a mobile phone, in the case where the mobile phone needs to determine whether the type of population to which the user belongs is the target population type based on the fused interaction data, the touch screen data in the sample interaction data may be stored when multiple users use terminal devices such as tablet computers and smart watches, and the sizes of the touch screens of tablet computers, smart watches and mobile phones are different. For example, the length of a user's sliding on the tablet display may be significantly longer than the length of a user's sliding on the mobile phone display, and the length of a user's sliding on the smart watch display may be significantly shorter than the length of a sliding on the mobile phone display. Therefore, splicing the touch screen data in the extended interaction data of the first age group obtained based on the sample interaction data to the touch screen data in the interaction data will make the touch screen data in the extended interaction data incompatible with the touch screen data in the interaction data. In view of this, the terminal device can adaptively migrate the sample interaction data according to the type of terminal device currently used by the user, for example, proportionally increase or decrease the sliding length data on the display screen in the touch screen data in the sample interaction data.
[0223] Combine the following Figure 7 , a method for a terminal device to determine whether the population type to which the user belongs is a target population type when the user is not a first type user or the first biometric data is not obtained is further described.
[0224] Figure 7 The process diagram of the crowd type identification method 700 provided in the embodiment of the present application when the user is not a first type of user or the first biometric data is not obtained. Figure 7 As shown, the sample interaction data includes sensor data, touch screen data, and application type data of multiple users using the terminal device in the historical time period. The terminal device can obtain sensor feature data of the sensor data, touch screen feature data of the touch screen data, and application type feature data of the application type data through the age model, and the feature data can be a feature vector or a feature matrix. Then, the terminal device clusters the sensor feature data, touch screen feature data, and application type feature data respectively to obtain extended interaction data for multiple age groups.
[0225] like Figure 7As shown, the interaction data includes the sensor data, touch screen data and application type data of the terminal device currently used by the user. The sensor data, touch screen data and application type data in the interaction data are respectively input into the trained deep learning convolution model to obtain sensor feature data, touch screen feature data and application type feature data. The feature data can be a feature vector or a feature matrix. Then, the sensor feature data, touch screen feature data and application type feature data are respectively classified into the extended interaction data of multiple age groups through the KNN classification method.
[0226] Optionally, the terminal device can perform dimensionality reduction processing on the sensor feature data, touch screen feature data and application type feature data in the extended interaction data of multiple age groups, and solve the orthogonal common features of the sensor feature data after dimensionality reduction, the orthogonal common features of the touch screen feature data after dimensionality reduction and the orthogonal common features of the application type feature data after dimensionality reduction. Then, the terminal device tile splices the orthogonal common features of the sensor feature data after dimensionality reduction in the extended interaction data of the first age group with the sensor feature data in the interaction data; tile splices the orthogonal common features of the touch screen feature data after dimensionality reduction in the extended interaction data of the first age group with the touch screen feature data in the interaction data; tile splices the orthogonal common features of the application type feature data after dimensionality reduction in the extended interaction data of the first age group with the application type feature data in the interaction data, and obtains fused interaction data. The fused interaction data can also be called code feature data. Finally, the terminal device determines whether the type of population to which the user belongs is a minor based on the fused interaction data and / or biometric data. The biometric data may include the first biometric data and / or the third biometric data. Optionally, when the first biometric data and the third biometric data are not obtained, the terminal device determines whether the type of people the user belongs to is a minor based on the fused interaction data; when the first biometric data includes at least one of facial data, voice data or breathing data, and / or when the third biometric data is obtained, the terminal device can determine whether the type of people the user belongs to is a minor based on the fused interaction data and the biometric data, or based on the biometric data.
[0227] Optionally, when the terminal device determines whether the type of population to which the user belongs is a minor based on the fused interaction data and the biological data, the terminal device may first fuse the fused interaction data with the feature data of the biological data to obtain spliced feature data; the spliced feature data is input into the interaction-biological model to determine whether the type of population to which the user belongs is a minor type. Among them, the interaction-biological model can be recorded as model B, and the interaction-biological model can include a face-human speech sound model, a breathing-human speech sound model, a weak model, and a biological-interaction model, wherein the face-human speech sound model or the breathing-human speech sound model can determine the first age information based on the feature data of the first biological data; the weak model can determine the user's habit information of using the terminal device based on the fused interaction data; the biological-interaction model can determine whether the type of population to which the user belongs is the target population type based on the first age information and the user's habit information of using the terminal device. Among them, the habit information can also be a feature vector or a feature matrix.
[0228] Figure 8 A schematic diagram of another crowd type identification method 800 provided in an embodiment of the present application. Figure 8 As shown, when the terminal device obtains the first biometric data, it determines whether the user is a first type user based on the first biometric data. Specifically, the terminal device can input the first biometric data into the machine owner identification model, and the machine owner identification model outputs that the user is a first type user or not a first type user. Then, the terminal device can determine the feedback delay of the current service based on the service requirements of the current service, and determine whether the feedback delay of the current service is greater than or equal to the second preset threshold. In the case that the user is a first type user, the user's historical usage data and second biometric data are obtained; in the case that the feedback delay of the current service is greater than or equal to the second preset threshold, the third biometric data is obtained. The terminal device determines whether the type of population to which the user belongs is the target population type based on the historical usage data and the biometric data, and the biometric data includes the second biometric data, and in the case that the first biometric data includes at least one of face data, sound data or breathing data, the biometric data also includes the first biometric data; in the case that the feedback delay of the current service is greater than or equal to the second preset threshold, the biometric data also includes the third biometric data.
[0229] Specifically, the terminal device can determine whether the type of population the user belongs to is the target population type based on historical usage data and biological data in the following way: on the one hand, the terminal device inputs the facial data in the biological data into the face-image model, and the face-image model outputs a facial feature vector; the terminal device inputs the sound data or breathing data in the biological data into the audio model, and the audio model outputs a sound feature vector or a breathing feature vector. Then, in the case where the biological data includes sound data and face data, the terminal device inputs the face feature vector and the sound feature vector into the face-human speech sound model, and the face-human speech sound model can output the first age information; in the case where the biological data includes respiration data and face data, the terminal device inputs the face feature vector and the respiration feature vector into the face-respiration model, and the face-respiration model can output the first age information; in the case where the biological data includes face data, the terminal device can input the face feature vector into at least one of the face model, the face-human speech sound model or the face respiration model to obtain the first age information; in the case where the biological data includes sound data, the terminal device can input the sound feature vector into the face-human speech sound model to obtain the first age information; in the case where the biological data includes respiration data, the terminal device can input the respiration feature vector into the face-respiration model to obtain the first age information; in the case where the biological data includes sound data and respiration data, the terminal device can input the sound feature vector into the face-human speech sound model to obtain the first age information. The first age information can be the age group to which the user belongs and the probability that the user belongs to the age group, or it can be a feature vector or a feature matrix. On the other hand, the terminal device can obtain the feature vector of the historical usage data through the deep learning convolution model, and then input the feature vector of the historical usage data into the user portrait model. The user portrait model can output the user's living habit information and / or the habit information of using the terminal device based on the historical usage data, such as the types of food that the user buys most often, the types of web pages that the user browses most often, etc. The habit information can also be a feature vector or a feature matrix. Finally, the terminal device can determine whether the type of population to which the user belongs is the target population type based on the first age information and the habit information. Specifically, the terminal device can input the first age information and the habit information into the age-habit model, and the age-habit model outputs whether the type of population to which the user belongs is the target population type.
[0230] like Figure 8 As shown, when the user is not a first type user, the terminal device can determine whether the type of population to which the user belongs is a target population type based on the biometric data and / or the interaction data. When the first biometric data includes at least one of face data, sound data, or breathing data, the biometric data includes the first biometric data; when the feedback delay of the current service is greater than or equal to the second preset threshold, the biometric data also includes the third biometric data.
[0231] Optionally, when the first biometric data includes at least one of face data, voice data or breathing data, and / or the feedback delay of the current service is greater than or equal to a second preset threshold, the terminal device determines whether the type of the crowd to which the user belongs is the target crowd type based on the biometric data. It should be understood that the specific implementation method of the terminal device determining whether the type of the crowd to which the user belongs is the target crowd type based on the biometric data is the same as the case where the user is a first type of user, which can be referred to above and will not be repeated here.
[0232] When the first biometric data does not include face data, voice data, and breathing data and the feedback delay of the current service is less than the second preset threshold, the terminal device determines whether the type of population to which the user belongs is the target population type based on the interaction data. Specifically, the terminal device inputs the sensor data in the interaction data into the inertial measurement unit (IMU) model and outputs the feature vector of the sensor data; the terminal device inputs the touch screen data in the interaction data into the sliding model and outputs the feature vector of the touch screen data; the terminal device inputs the application type data in the interaction data into the application (APP) model and outputs the feature vector of the application type data. The IMU model, the sliding model, and the APP model can be deep learning convolution models. Then, the terminal device inputs the feature vector of the sensor data, the feature vector of the touch screen data, and the feature vector of the application type data into the weak model, and the weak model can input whether the type of population to which the user belongs is the target population type or not the target population type.
[0233] It can be understood that the face data in the embodiment of the present application may include face image data obtained by the terminal device when the user uses face unlocking of the terminal device; the voice data and breathing data of the user can be cached in the terminal device when the user uses voice calls, video calls, voice assistants and other services, or can be obtained when the terminal device needs to determine whether the type of group of people to which the user belongs is the target group type; the sensor data can be the data cached by the terminal device when the user uses the terminal device to call some APPs, or can be the data obtained when the terminal device needs to determine whether the type of group of people to which the user belongs is the target group type; the touch screen data can be the data collected by the terminal device when the user touches the display screen of the terminal device The data is stored in the cache in the form of a sliding window. For example, the amount of touch screen data stored in the terminal device is constant. As new touch screen data is generated, the touch screen data farthest from the current moment will be deleted accordingly. When the terminal device needs to determine whether the type of population the user belongs to is the target population type based on the touch screen data, the touch screen data stored in the cache is called, that is, the countdown N data before the touch screen data is obtained is called. The application type data can be collected when the user uses the application of the terminal device and stored in the cache in the form of a sliding window. When the terminal device needs to determine whether the type of population the user belongs to is the target population type based on the application type data, the cached application type data is called.
[0234] As an optional embodiment, when the crowd type identification switch is turned on, the terminal device periodically or when the current service of the terminal device is a preset service type, the terminal device determines whether the crowd type to which the user belongs is the target crowd type. It should be understood that the crowd type identification switch can be a selection button in the system setting interface of the terminal device, such as Fig. 9 As shown in the selection button 901 in FIG. When the user clicks the selection button 901, the crowd type recognition switch can be turned on or off.
[0235] It should be understood that the various models involved in the embodiments of the present application, such as face-image models, face models, human speech sound models, biometric fusion models, biometric-user portrait fusion models, age models, weak models, etc., can be neural network models, deep learning convolution models and other models, which can all be obtained through deep learning. The present application does not impose any specific restrictions on this.
[0236] It should be understood that the various models involved in the embodiments of the present application, such as face-image model, face model, human speech sound model, biometric fusion model, biometric-user portrait fusion model, age model, weak model, etc., can also be named by other names. Any model that can achieve the corresponding effect can be used as the above-mentioned model, and the present application does not make any specific limitation.
[0237] It should be understood that the order of execution of the above methods does not necessarily mean the order in which they are executed. The order in which each process is executed should be determined by its function and internal logic.
[0238] Combination of the above Figures 3 to 9 , describes in detail the crowd type identification method of the embodiment of the present application, and the following is combined with Fig.10 and Fig.11 , a crowd type identification device according to an embodiment of the present application is described in detail. The crowd type identification device includes a module or unit for executing each part of the above embodiment. The module or unit can be software, hardware, or a combination of software and hardware. The following only briefly illustrates the crowd type identification device. For the implementation details of the solution, please refer to the description of the aforementioned method embodiment, which will not be repeated below.
[0239] Fig.10 Schematic block diagram of a crowd type identification device 1000 provided in an embodiment of the present application. Fig.10 As shown, the device 1000 includes: an acquisition module 1001 and a processing module 1002 .
[0240] In a possible implementation manner, the apparatus 1000 may be used to implement the steps corresponding to the terminal device in the above method embodiment.
[0241] The acquisition module 1001 is used to determine whether the user is a first type of user based on the first biometric data when the user is currently using the device 1000, the first biometric data including at least one of voice data, breathing data, fingerprint data or face data, and the first type of user is a user whose usage time and / or usage frequency of the device 1000 exceeds a first preset threshold; when the user is a first type of user, the historical data of the device 1000 is acquired, the historical data including the historical usage data of the application on the device 1000 and the second biometric data of the user when the device 1000 is historically used, the second biometric data including at least one of voice data, breathing data or face data; the processing module 1002 is used to determine whether the type of population to which the user belongs is a target population type based on the historical data.
[0242] Optionally, the processing module 1002 is specifically used to: when the first biological data includes at least one of sound data, breathing data or face data, determine whether the type of population the user belongs to is the target population type based on historical data and the first biological data.
[0243] Optionally, the processing module 1002 is specifically used to: fuse the first biometric data and the second biometric data to obtain fused biometric data; determine the user's first age information based on the fused biometric data; and determine whether the user's population type is the target population type based on the first age information and historical usage data.
[0244] Optionally, the target population type is a minor type; the processing module 1002 is also used to: determine the user's second age information based on historical usage data; the processing module 1002 is specifically used to: determine whether the population type to which the user belongs is a minor type based on the first age information and the second age information.
[0245] Optionally, the processing module 1002 is specifically used for: when the fused biological data includes face data and voice data, inputting the fused biological data into a face-human speech voice model to obtain first age information, the face-human speech voice model is obtained by weighted fusion of the output result of the face model and the output result of the human speech voice model; or, when the fused biological data includes face data and breathing data, inputting the fused biological data into a face-breathing model to obtain the first age information, the face-breathing model is obtained by weighted fusion of the output result of the face model and the output result of the breathing model; or, when the fused biological data includes face data, voice data and breathing data, inputting the face data and voice data in the fused biological data into the face-human speech voice model to obtain the first age information.
[0246] Optionally, the facial data is valid facial data that has been screened.
[0247] Optionally, the acquisition module 1001 is also used to: when the first biometric data is not acquired, acquire the interaction data of the user when currently using the device 1000, the interaction data including at least one of sensor data, touch screen data or application type data; the processing module 1002 is also used to: based on the interaction data, determine whether the type of population to which the user belongs is the target population type.
[0248] Optionally, the acquisition module 1001 is also used to: when the user is not a first type of user, obtain the interaction data of the user when currently using the device 1000, the interaction data including at least one of sensor data, touch screen data or application type data; the processing module 1002 is also used to: determine whether the type of population to which the user belongs is the target population type based on the interaction data and / or the first biological data.
[0249] Optionally, the processing module 1002 is also used to: based on the interaction data, select extended interaction data of the first age group from extended interaction data of multiple age groups, the extended interaction data of the multiple age groups are obtained by clustering sample interaction data, the sample interaction data including at least one of sensor data, touch screen data or application type data of multiple users using the device 1000 in a historical time period; fuse the interaction data and the extended interaction data of the first age group to obtain fused interaction data; the processing module 1002 is specifically used to: determine whether the user is the target population type based on the fused interaction data.
[0250] Optionally, the processing module 1002 is further used to: use the age model to extract features from the sample interaction data to obtain feature data; and perform cluster analysis on the feature data to obtain extended interaction data for multiple age groups.
[0251] Optionally, the processing module 1002 is also used to: perform dimensionality reduction processing on the interaction data to obtain reduced dimensionality interaction data; perform dimensionality reduction processing on the extended interaction data of the first age group to obtain reduced dimensionality extended interaction data; the processing module 1002 is specifically used to: fuse the reduced dimensionality interaction data and the reduced dimensionality extended interaction data to obtain fused interaction data.
[0252] Optionally, the interaction data includes the reciprocal N pieces of touch screen data before the moment when the interaction data is acquired, where N is a positive integer.
[0253] Optionally, the interaction data includes M pieces of filtered valid touch screen data, where M is a positive integer.
[0254] Optionally, the processing module 1002 is also used to: determine the time interval between adjacent touch screen data in the N touch screen data before the moment of acquiring the interaction data based on the generation time of each touch screen data in the countdown N touch screen data before the moment of acquiring the interaction data, where N is a positive integer greater than or equal to M; determine the substitution time node based on the time interval; and determine the touch screen data after the substitution time node that is closest to the moment of acquiring the interaction data among the substitution time nodes as M valid touch screen data.
[0255] Optionally, the processing module 1002 is also used to: determine the feedback delay based on the business requirements of the current business of the device 1000; when the feedback delay is greater than or equal to a second preset threshold, obtain the third biometric data of the user using the device 1000 within a preset time period, the third biometric data including at least one of sound data, breathing data or face data; the processing module 1002 is specifically used to: determine whether the type of population to which the user belongs is the target population type based on historical data and the third biometric data.
[0256] It should be understood that the device 1000 here is embodied in the form of a functional module. The term "module" here may refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a proprietary processor or a group processor, etc.) and a memory for executing one or more software or firmware programs, a merged logic circuit and / or other suitable components that support the described functions. In an optional example, those skilled in the art can understand that the device 1000 can be specifically a terminal device in the above-mentioned embodiment, and the device 1000 can be used to execute the various processes and / or steps corresponding to the terminal device in the above-mentioned method embodiment. To avoid repetition, it will not be repeated here.
[0257] The above-mentioned apparatus 1000 has the function of implementing the corresponding steps executed by the terminal device in the above-mentioned method; the above-mentioned function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned function.
[0258] In the embodiments of the present application, Fig.10 The device 1000 in the embodiment may also be a chip, such as a SOC. Correspondingly, the processing module 1002 may be a transceiver circuit of the chip, which is not limited here.
[0259] Fig.11 A schematic block diagram of another crowd type identification device 1100 provided in an embodiment of the present application is shown. The device 1100 includes a processor 1101, a transceiver 1102, and a memory 1103. The processor 1101, the transceiver 1102, and the memory 1103 communicate with each other through an internal connection path, the memory 1103 is used to store instructions, and the processor 1101 is used to execute the instructions stored in the memory 1103 to control the transceiver 1102 to send signals and / or receive signals.
[0260] It should be understood that the device 1100 can be specifically a terminal device in the above embodiment, and can be used to execute the various steps and / or processes corresponding to the terminal device in the above method embodiment. Optionally, the memory 1103 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type. The processor 1101 may be used to execute instructions stored in the memory, and when the processor 1101 executes the instructions stored in the memory, the processor 1101 is used to execute the various steps and / or processes of the above method embodiment. The transceiver 1102 may include a transmitter and a receiver, the transmitter may be used to implement the various steps and / or processes corresponding to the above transceiver for performing the sending action, and the receiver may be used to implement the various steps and / or processes corresponding to the above transceiver for performing the receiving action.
[0261] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.
[0262] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.
[0263] The present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to implement the method shown in the above method embodiment.
[0264] The present application also provides a computer program product, which includes a computer program (also referred to as code or instruction). When the computer program runs on a computer, the computer can execute the method shown in the above method embodiment.
[0265] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0266] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0267] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0268] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0269] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.
[0270] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0271] The above is only a specific implementation of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the embodiments of the present application, which should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be based on the protection scope of the claims.
Claims
1. A method for identifying crowd types, characterized in that: include: In the case of obtaining first biometric data of a user when currently using a terminal device, determining whether the user is a first type of user based on the first biometric data, the first biometric data including at least one of voice data, breathing data, fingerprint data or face data, and the first type of user is a user whose usage time or number of usages of the terminal device exceeds a first preset threshold; When the user is a user of the first type, acquiring historical data of the terminal device, the historical data including historical usage data of the application on the terminal device and second biometric data of the user when the terminal device was historically used, the second biometric data including at least one of voice data, breathing data or face data; Based on the historical data, it is determined whether the type of population to which the user belongs is a target population type.
2. The method according to claim 1, characterized in that The determining, based on the historical data, whether the type of population to which the user belongs is a target population type includes: In the case where the first biological data includes at least one of voice data, breathing data or face data, it is determined whether the type of population to which the user belongs is the target population type based on the historical data and the first biological data.
3. The method according to claim 2, characterized in that The determining, based on the historical data and the first biological data, whether the type of population to which the user belongs is the target population type includes: fusing the first biological data and the second biological data to obtain fused biological data; Determining first age information of the user based on the fused biometric data; Based on the first age information and the historical usage data, it is determined whether the population type to which the user belongs is the target population type.
4. The method according to claim 3, characterized in that The target population type is the type of minors; The method further comprises: determining second age information of the user based on the historical usage data; The determining whether the type of population to which the user belongs is the target population type includes: Based on the first age information and the second age information, it is determined whether the type of population to which the user belongs is the minor type.
5. The method according to claim 3 or 4, characterized in that: The determining the first age information of the user based on the fused biometric data includes: In the case where the fused biometric data includes face data and voice data, the fused biometric data is input into a face-human speech voice model to obtain the first age information, wherein the face-human speech voice model is obtained by weighted fusion of an output result of the face model and an output result of the human speech voice model; or In the case where the fused biometric data includes face data and respiration data, the fused biometric data is input into a face-respiration model to obtain the first age information, wherein the face-respiration model is obtained by weighted fusion of an output result of the face model and an output result of the respiration model; or In the case where the fused biometric data includes face data, voice data and breathing data, the face data and voice data in the fused biometric data are input into the face-human speech voice model to obtain the first age information.
6. The method according to claim 5, characterized in that The facial data is valid facial data that has been screened.
7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: If the first biometric data is not obtained, obtaining interaction data of the user when currently using the terminal device, the interaction data including at least one of sensor data, touch screen data or application type data; Based on the interaction data, it is determined whether the type of population to which the user belongs is the target population type.
8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: In a case where the user is not a user of the first type, acquiring interaction data of the user when currently using the terminal device, the interaction data including at least one of sensor data, touch screen data, or application type data; Based on the interaction data and / or the first biological data, it is determined whether the type of population to which the user belongs is the target population type.
9. The method according to claim 7 or 8, characterized in that: Before determining whether the type of population to which the user belongs is the target population type, the method further includes: Based on the interaction data, selecting extended interaction data of a first age group from extended interaction data of multiple age groups, wherein the extended interaction data of the multiple age groups are obtained by clustering sample interaction data, wherein the sample interaction data includes at least one of sensor data, touch screen data, or application type data of a terminal device used by multiple users in a historical time period; fusing the interaction data with the extended interaction data of the first age group to obtain fused interaction data; The determining whether the type of population to which the user belongs is the target population type includes: Based on the fused interaction data, determine whether the user is the target population type.
10. The method according to claim 9, characterized in that Before selecting the extended interaction data of the first age group from the extended interaction data of the plurality of age groups, the method further includes: Using the age model, extracting features from the sample interaction data to obtain feature data; Cluster analysis is performed on the characteristic data to obtain extended interaction data of the multiple age groups.
11. The method according to claim 9 or 10, characterized in that: Before fusing the interaction data with the extended interaction data of the first age group, the method further includes: Performing dimensionality reduction processing on the interaction data to obtain dimensionality-reduced interaction data; Performing dimensionality reduction processing on the extended interaction data of the first age group to obtain dimensionality-reduced extended interaction data; The fusing the interaction data with the extended interaction data of the first age group includes: The dimension reduction interaction data and the dimension reduction extension interaction data are fused to obtain the fused interaction data.
12. The method according to any one of claims 7 to 11, characterized in that The interaction data includes the reciprocal N pieces of touch screen data before the moment when the interaction data is acquired, where N is a positive integer.
13. The method according to any one of claims 7 to 11, characterized in that The interaction data includes M pieces of screened valid touch screen data, where M is a positive integer.
14. The method according to claim 13, characterized in that The method further comprises: Determine, according to the generation time of each touch screen data in the reciprocal N touch screen data before the moment of acquiring the interaction data, the time interval between adjacent touch screen data in the N touch screen data, where N is a positive integer greater than or equal to M; Determine a time node for substitution according to the time interval; The touch screen data after the person-changing time node closest to the moment of acquiring the interaction data among the person-changing time nodes are determined as the M pieces of valid touch screen data.
15. The method according to any one of claims 1 to 14, characterized in that Before determining whether the group type to which the user belongs is the target group type, the method further includes: Determining a feedback delay based on a service requirement of a current service of the terminal device; When the feedback delay is greater than or equal to a second preset threshold, obtaining third biometric data of the user using the terminal device within a preset time period, the third biometric data including at least one of voice data, breathing data or face data; The determining, based on the historical data, whether the type of population to which the user belongs is a target population type includes: Based on the historical data and the third biological data, it is determined whether the type of population to which the user belongs is the target population type.
16. A crowd type recognition device, characterized in that: Comprising means for performing the method as claimed in any one of claims 1 to 15.
17. A crowd type recognition device, characterized in that: include: A processor, wherein the processor is coupled to a memory, wherein the memory is used to store a computer program, and when the processor calls the computer program, the device is caused to execute the method according to any one of claims 1 to 15.
18. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program comprising instructions for implementing the method according to any one of claims 1 to 15.
19. A computer program product, characterized in that The computer program product includes a computer program code, and when the computer program code is executed on a computer, the computer is enabled to implement the method according to any one of claims 1 to 15.
Citation Information
Cited By
Minor identification method, electronic device, and computer-readable storage medium
WO2026113554A1
