Crowd recognition method, device and storage medium

Abstract

The present application provides a crowd recognition method, device and storage medium, which relate to the terminal technology field and the artificial intelligence field, and also to the technical fields of intelligent perception, intelligent control, intelligent recommendation, etc. The method includes: after the crowd recognition function of the electronic device is turned on, the camera operates in the first mode. When the electronic device detects that there is a human face within the camera range, it controls the camera to operate in the second mode. The electronic device obtains the image captured by the camera in the second mode, identifies the crowd category of the person in the image, and executes preset operations corresponding to the crowd category, such as content recommendation, parameter setting, reminder to rest, etc., to realize the function of intelligent perception of specific groups of people and improve the user experience of users from different groups.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a crowd identification method, device and storage medium. Background Art

[0002] With the popularization of smart terminals, the number of people using terminals has grown rapidly. Users belong to different age groups and have their own special group characteristics, and there are differentiated usage problems: for example, the elderly have deteriorating vision and hearing, and cannot see the text on the mobile phone screen clearly, cannot hear the sound clearly, often click on the wrong advertisements, or encounter telecommunications fraud, etc.; for example, children are very sensitive, and the screen display is too bright, the sound is too loud, causing irreversible effects, and lack of self-control leads to excessive use of the phone, etc.; for example, teenagers play games and watch short videos in the name of learning and taking online classes, and the content of video applications varies, which may affect their physical and mental health.

[0003] Currently, there is no solution for terminals to perceive crowds in related technologies. How terminals can achieve crowd recognition is an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present application provide a crowd identification method, device and storage medium, which can realize the function of intelligently sensing a specific group of people and improve the user experience of different groups of users.

[0005] In a first aspect, an embodiment of the present application proposes a crowd recognition method, which is applied to an electronic device, in which a camera of the electronic device operates in a first mode; the electronic device obtains a first image captured by the camera in the first mode, and detects whether there is a human face in the first image; if the electronic device detects a human face in the first image, the camera is controlled to switch from the first mode to the second mode; the electronic device obtains a second image captured by the camera in the second mode, and identifies the crowd category of the person in the second image; the electronic device identifies that the crowd category of the person is the first category, and controls the execution of a preset operation corresponding to the first category.

[0006] In this solution, the camera continuously captures images in the first mode. If a person's face is detected in the image, it switches to the second mode, continuously captures images in the second mode, and identifies the crowd category corresponding to the person's face in the image, and then performs different operations based on the crowd category, realizing the function of intelligently sensing the crowd category, providing different needs for users of different groups, and improving the user experience. In addition, by acquiring the first image captured by the camera in the first mode, detecting whether there is a person's face, and then switching the camera mode to acquire the second image captured by the camera in the second mode, and identifying the crowd category of the person, the power consumption of the device can be reduced to a certain extent.

[0007] As an example, a resolution of the first image is smaller than a resolution of the second image, and / or a frame rate of the first image is smaller than a frame rate of the second image.

[0008] As an example, the first category includes children, teenagers, and the elderly.

[0009] As an example, the first category includes men and women.

[0010] In an optional embodiment of the first aspect, before the camera of the electronic device operates in the first mode, the method further includes: responding to a first operation of enabling a crowd recognition function.

[0011] In this solution, the conditions for turning on the camera of the device are defined to trigger the device to execute the solution of intelligently sensing the crowd category.

[0012] In an optional embodiment of the first aspect, before the camera of the electronic device operates in the first mode, the method also includes: detecting that the state of the electronic device satisfies a first condition; the first condition includes at least one of the following: the screen state of the electronic device is a bright screen state; the electronic device is unlocked; the time difference between a light signal emitted by a proximity light sensor of the electronic device and a reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the screen of the electronic device is facing a preset direction; the preset application of the electronic device is turned on.

[0013] In this solution, the conditions for turning on the camera of the device are further limited. In addition to the user manually turning on the crowd recognition function, the first condition is added to prevent the camera from continuously collecting images when it is not necessary, thereby reducing the power consumption of the device. When it is not necessary, it can be understood as any scenario in which the camera cannot capture the face of a person.

[0014] In an optional embodiment of the first aspect, the electronic device is a foldable device, the foldable device includes an inner screen and an outer screen, the inner screen is correspondingly provided with a first camera, and the outer screen is correspondingly provided with a second camera; the camera of the electronic device operates in a first mode, including: detecting that the outer screen of the electronic device is in a bright screen state and the electronic device is in a folded state, and controlling the second camera to operate in the first mode; or, detecting that the inner screen of the electronic device is in a bright screen state and the electronic device is in an unfolded state, and controlling the first camera to operate in the first mode.

[0015] This solution can be applied to foldable devices. If the device is in a folded or unfolded state and the corresponding screen (inner screen or outer screen) is in a bright screen state, the camera on the bright screen can be controlled to turn on to detect whether there is a human face within the camera range, thereby triggering the recognition of the person's crowd category.

[0016] In an optional embodiment of the first aspect, before controlling the first camera to operate in the first mode, or controlling the second camera to operate in the first mode, the method also includes: detecting that the state of the electronic device satisfies a second condition; the second condition includes at least one of the following: the electronic device is unlocked; the time difference between a light signal emitted by a proximity light sensor of the electronic device and a reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than a third threshold; the inner screen or outer screen of the electronic device is facing a preset direction; the preset application of the electronic device is turned on.

[0017] In this solution, the conditions for turning on the camera of a foldable device are further limited. In addition to the corresponding screen lighting up when the user turns on the crowd recognition function and the electronic device screen is in a folded or unfolded state, a second condition is added to avoid the camera from continuously collecting the first image when it is not necessary, thereby reducing the power consumption of the device.

[0018] In an optional embodiment of the first aspect, the method also includes: when the second camera is operating in the first mode, if it is detected that the electronic device changes from a folded state to an unfolded state, the electronic device controls the first camera to operate in the first mode and turns off the second camera; when the first camera is operating in the first mode, if it is detected that the electronic device changes from an unfolded state to a folded state, the electronic device controls the first camera to turn off and controls the second camera to operate in the first mode.

[0019] In this solution, when other conditions remain unchanged, if the user changes the physical state of the device screen, such as from a folded state to an unfolded state, or from an unfolded state to a folded state, the first image can be continuously captured by switching the camera, so that the device screen can also realize the function of intelligently sensing crowd categories in the new physical state.

[0020] In an optional embodiment of the first aspect, the camera of the electronic device operates in a first mode, including: a perception module of the electronic device sends a first indication to a second processing module of the electronic device, the first indication being used to instruct the second processing module to detect whether there is a human face within the camera range; the second processing module sends a first shooting instruction to the camera; and the camera operates in the first mode in response to the first shooting instruction.

[0021] In an optional embodiment of the first aspect, the electronic device obtains a first image captured by the camera in the first mode, and detects whether there is a human face in the first image, including: a second processing module of the electronic device obtains the first image captured by the camera in the first mode, and detects whether there is a human face in the first image.

[0022] In an optional embodiment of the first aspect, if the electronic device detects a human face in the first image, the camera is controlled to switch from the first mode to the second mode, including: if the second processing module of the electronic device detects that there is a human face in the first image, the second processing module sends a first message to the perception module of the electronic device, and the first message is used to notify the perception module that there is a human face within the camera range; the perception module sends a second indication to the first processing module of the electronic device, and the second indication is used to instruct the first processing module to identify the crowd category of the person in the second image; the first processing module sends a second shooting instruction to the camera in response to the second indication, and the second shooting instruction is used to instruct the camera to operate in the second mode.

[0023] In an optional embodiment of the first aspect, the electronic device obtains a second image captured by the camera in the second mode, and identifies the crowd category of the person in the second image, including: a first processing module of the electronic device obtains the second image captured by the camera in the second mode, and identifies the crowd category of the person in the second image.

[0024] In an optional embodiment of the first aspect, if the electronic device recognizes that the crowd category of the person in the second image is the first category, control executes a preset operation corresponding to the first category, including: if the first processing module of the electronic device recognizes that the crowd category of the person in the second image is the first category, the first processing module sends a second message to the perception module of the electronic device, and the second message is used to indicate that the person in the second image is the first category; the perception module sends a third indication to the target application of the electronic device, and the third indication is used to indicate that the person is the first category; the target application controls the execution of the preset operation corresponding to the first category.

[0025] In an optional embodiment of the first aspect, the second processing module of the electronic device detects a state of the electronic device.

[0026] The above-mentioned optional embodiments illustrate the interaction process between the underlying modules of the electronic device to realize the function of the device intelligently sensing the category of the crowd, provide different needs for different groups of people, and improve the user experience.

[0027] In an optional embodiment of the first aspect, the preset operations include at least one of the following: adjusting the volume; adjusting the brightness; sending a first message, the first message is used to prompt whether to change the font size, adjust the volume or brightness; sending a second message, the second message is used to prompt installation risks or traffic risks; sending a third message to a first external device, the third message is used for payment verification; sending a fourth message, the fourth message is used to remind to take a break; sending a fifth message to a second external device, the fifth message is used to indicate the category of people currently using the electronic device, so that the second external device pushes content corresponding to the category of people to the electronic device; identity authentication.

[0028] In this solution, different groups of people can correspond to different operations to meet the machine usage needs of different groups of people.

[0029] In a second aspect, an embodiment of the present application provides an electronic device, comprising: a camera, a memory and a processor, wherein the camera is used to capture images with different frame rates and / or resolutions, and the processor is used to call a computer program in the memory to execute a method as described in any one of the first aspects.

[0030] In an optional embodiment of the second aspect, the processor includes a first processing module and a second processing module, the power consumption of the first processing module is higher than that of the second processing module; the second processing module is used to detect whether there is a human face within the camera range; and the first processing module is used to identify the crowd category of the people within the camera range.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a unit, a module or a circuit for executing any method as described in the first aspect.

[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in any one of the first aspects.

[0033] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute a method as described in any one of the first aspects.

[0034] In a sixth aspect, a computer program product comprises a computer program, and when the computer program is executed, the computer executes the method as described in any one of the first aspect.

[0035] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a crowd identification method provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of an interface provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0039] Figure 4A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of the structure of a SoC provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0042] Figure 7 A flow chart of a crowd identification method provided in an embodiment of the present application;

[0043] Figure 8 A schematic diagram of a crowd identification method provided in an embodiment of the present application;

[0044] Fig. 9 A flow chart of a crowd identification method provided in an embodiment of the present application;

[0045] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0046] Fig.11 A schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0047] Fig.12 A schematic diagram of the structure of a foldable screen mobile phone provided in an embodiment of the present application;

[0048] Fig.13 A flow chart of a crowd identification method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with substantially the same functions and effects. For example, the first image and the second image are only used to distinguish between images with different frame rates and / or resolutions, and their order is not limited. For another example, the first indication and the second indication are only used to distinguish between different indications. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit them to be different.

[0050] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0051] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following (kind / piece)" or similar expressions refers to any combination of these items, including any combination of single items (kind / piece) or plural items (kind / piece). For example, at least one of a, b or c (kind / piece) can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0052] Some of the terms used in this application are explained below to facilitate understanding by those skilled in the art.

[0053] Frame rate refers to the number of images captured or transmitted by a camera in one second, usually expressed in fps (frames per second). In the embodiment of the present application, the camera uses a first frame rate to capture / transmit images in the first mode and a second frame rate to capture / transmit images in the second mode, and the first frame rate is less than the second frame rate. In some embodiments, the resolution of the image captured in the first mode is less than the resolution of the image captured in the second mode.

[0054] Resolution, also known as image resolution, refers to the amount of information stored in an image, which is the number of pixels per inch of the image. The units of resolution include: dpi (dots per inch), ppi (pixels per inch), etc.

[0055] A lightweight neural network is a lighter model that has performance that is no worse than that of a heavier model, thereby realizing a hardware-friendly neural network. The weight here usually refers to the scale or number of parameters of the model. Commonly used lightweight neural network technologies include: distillation, pruning, quantization, weight sharing, low-rank decomposition, lightweight attention module, dynamic network architecture / training method, lighter network architecture design, etc., which are not limited to this embodiment of the present application.

[0056] With the popularization of smart terminals, the number of groups using terminals has grown rapidly. According to age, user groups can be roughly divided into: children, teenagers, adults and the elderly, and according to gender, user groups can be divided into: men and women. Different user groups have different needs when using mobile phones.

[0057] In terms of information recommendation, the information content that people of different ages or genders are interested in varies greatly. For third-party applications, the usual practice is to recommend information based on the user's historical browsing history. If the user is a new user of a third-party application, the initial information recommendation is random. As a certain amount of user interaction data is collected, accurate content recommendations are made based on the user interaction data.

[0058] In terms of time spent on the device, for example, short video applications currently provide a youth mode, which limits the time users spend watching short videos. Specifically, the device owner can manually set a password and time spent on the device to prevent teenagers from becoming addicted to the device. Another example is a game application. In order to prevent teenagers from becoming addicted to games, the game server sets a time period for teenagers to play games. For example, only on weekends from 8 to 9 p.m., teenage players can log in to the game server. However, some players may use adult accounts to play games.

[0059] In terms of parameter settings, there may be many inconveniences for vulnerable groups when using terminals: for example, in the system or third-party application interface, the font size of the mobile phone screen is too small, and the elderly or people with poor eyesight cannot see the text on the mobile phone screen clearly; for example, when playing videos or audio, the volume is too low, and the elderly cannot hear clearly and are not very good at using mobile phones; for example, when browsing pages, various advertising windows pop up, and the elderly cannot see clearly or do not read carefully, and click on the advertising windows, resulting in the installation of many unnecessary software, etc. Another example is that when children use mobile phones, the screen shows dawn, or the sound is too loud, which will cause irreversible effects.

[0060] In terms of information security, vulnerable groups, especially the elderly, often encounter telecommunications fraud, which is difficult to prevent. For example, they click on unknown links in chat groups, click on SMS links or send SMS content to others, or place orders for large amounts of goods or financial products on informal websites.

[0061] In response to the above problems, the embodiments of the present application provide a crowd recognition method, an electronic device, and a storage medium. The electronic device provided by the embodiments of the present application, if the conditions for turning on the camera are met, the camera operates in the first mode, obtains the first image captured by the camera in the first mode, and if there is a person's face in the first image, controls the camera to switch from the first mode to the second mode. The second image captured by the camera in the second mode is obtained, the crowd category of the person in the second image is identified, and the preset operation corresponding to the crowd category is executed, such as automatically setting system parameters such as volume and brightness, or limiting the machine usage time, or recommending content corresponding to the crowd category, etc. The above scheme realizes the function of the electronic device to intelligently perceive the crowd, and the user does not need to manually set the relevant parameters or inform the points of interest, etc., which can improve the user's machine experience.

[0062] The following specific embodiments are used to describe in detail the technical solution of the present application and how the technical solution of the present application solves the above technical problems. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0063] The following description will be given by taking a mobile phone as an example of an electronic device, which does not constitute a limitation on the embodiments of the present application.

[0064] Figure 1 A schematic diagram of a crowd identification method provided in an embodiment of the present application. Figure 1 As shown, in the crowd recognition scenario, if the conditions for turning on the camera are met, the front camera of the mobile phone can be triggered to capture a first image in a first mode. When it is detected that the first image contains a person's face, the front camera of the mobile phone is triggered to capture a second image in a second mode, and the crowd category of the person in the second image is identified as the first category, and then the preset operation corresponding to the first category is executed.

[0065] In this embodiment, the power consumption of the camera in the first mode is lower than that in the second mode. The resolution of the first image is lower than that of the second image, and / or the frame rate of the first image is lower than that of the second image. The first image includes one or more images, and the second image includes one or more images.

[0066] In some embodiments, the first category includes children, teenagers, and the elderly.

[0067] Exemplarily, if the mobile phone recognizes that the person in the second image is a child, at least one of the following operations can be performed: lowering the volume of the mobile phone, lowering the brightness of the mobile phone screen, popping up a window to remind you to take a break if the phone is used continuously for more than a preset time (for example, 30 minutes), answering a question to close the pop-up window (the question can be set by the owner), requiring a pop-up window for verification when starting the game (the game can be started after the owner has authenticated it), and pushing content that children are interested in (such as cartoons, etc.).

[0068] Exemplarily, if the mobile phone recognizes that the person in the second image is a teenager, at least one of the following operations can be performed: a pop-up window will be displayed to remind people to take a break if the game server is used continuously for more than a preset time, a pop-up window will be displayed to explain when accessing the game server during non-opening hours, and content that teenagers are interested in (such as popular science, positive content, etc.) will be pushed.

[0069] Exemplarily, if the mobile phone recognizes that the task in the second image is an elderly person, it can perform at least one of the following operations: increase the volume of the mobile phone, lower the brightness of the mobile phone screen, pop-up window to remind whether to change the font size or adjust the volume, pop-up window reminder when clicking a link or installing an application, pop-up window reminder when sending text messages, send a message to the guardian when the payment amount is greater than the preset amount (i.e., guardian verification is required), and push content corresponding to the elderly.

[0070] In some embodiments, the first category includes men and women. Exemplarily, if the mobile phone recognizes that the person in the second image is a man (or a woman), content that the man (or woman) is interested in may be pushed.

[0071] In some embodiments, the condition for turning on the camera includes: the crowd recognition function has been turned on. In one embodiment, in response to the first operation of turning on the crowd recognition function, the camera of the mobile phone runs in the first mode (background operation, for example, the mobile phone currently opens a third-party application and the camera runs in the background) to capture the above-mentioned first image. The first operation can be a click operation of the user in the system setting interface, and the first operation can also be a voice operation, which is not limited in the embodiments of the present application.

[0072] It should be noted that in the embodiments of the present application, unless otherwise specified, the camera of the mobile phone mainly refers to the front camera.

[0073] For example, Figure 2 This is a schematic diagram of an interface provided in an embodiment of the present application. Figure 2 As shown in (a) and (b), the user can choose to turn on or off the crowd recognition function in the setting interface of the system application. When it is detected that the crowd recognition function has been turned on on the mobile phone, the front camera of the mobile phone is triggered to capture the above-mentioned first image.

[0074] In some embodiments, the user can also choose to enable the crowd recognition function for all or some third-party applications in the system settings interface, such as Figure 2 As shown in (a), (c), and (d), the user turns on the crowd recognition function and selects application A and application B that support the crowd recognition function. When the user turns on application A or application B, the front camera of the mobile phone is triggered to capture the first image mentioned above.

[0075] In some embodiments, the user can also choose to turn on or off the crowd recognition function in the settings interface of the third-party application. When it is detected that the mobile phone has opened the third-party application and the crowd recognition function is turned on in the third-party application, the front camera of the mobile phone is triggered to capture the above-mentioned first image.

[0076] In an optional implementation, if the user has turned on the crowd recognition function and it is detected that the user has opened a preset third-party application, facial detection is started, and the facial detection includes detecting whether there is a human face and identifying the human crowd category. Exemplarily, each time the user opens a preset short video application or game application, the front camera of the mobile phone is triggered to capture a first image in the first mode. If the first image contains a human face, the mobile phone camera is triggered to capture a second image in the second mode, identify the human crowd category in the second image, and perform the preset operation corresponding to the crowd category. In some embodiments, the detection can be started each time the user opens a preset application, and no detection is performed during use.

[0077] In an optional implementation, if the user has turned on the crowd recognition function, the first level of facial detection is started, that is, whether there is a human face within the camera range. If a human face is detected within the camera range, and it is detected that the user has opened a preset third-party application, the second level of facial detection is started, that is, the crowd category of the person is identified. Exemplarily, after the user unlocks the phone, the front camera of the phone is triggered to capture a first image in the first mode to detect whether there is a human face within the camera range. If there is a human face within the camera range and the user opens a preset short video application or game application, the front camera of the phone is triggered to capture a second image in the second mode, identify the crowd category of the person in the second image, and execute the preset operation corresponding to the crowd category.

[0078] It should be noted that, in the embodiment of the present application, the third-party application that has been opened mainly refers to the user currently clicking on the third-party application, and the current interface is the interface of the third-party application, for example, the user is watching a short video, and the user is playing a game. The third-party application that has been opened does not include the third-party application entering the background running state after the user opens the third-party application.

[0079] In some embodiments, the condition for turning on the camera further includes a first condition, and the first condition includes at least one of the following:

[0080] The screen of the mobile phone is on.

[0081] The phone is unlocked;

[0082] The time difference between the light signal emitted by the proximity light sensor of the mobile phone and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal;

[0083] The detection data of the ambient light sensor of the mobile phone is greater than the third threshold;

[0084] The phone's screen faces the preset direction;

[0085] The phone's default app is open.

[0086] In one embodiment, in response to the first operation of turning on the crowd recognition function and satisfying the first condition, the camera of the mobile phone operates in the first mode to capture the above-mentioned first image. In this embodiment, if the crowd recognition function of the mobile phone is turned on and the first condition is satisfied, the camera of the mobile phone is triggered to continuously capture images. By adding the first condition, the mobile phone camera is prevented from continuously capturing images when it is not necessary, thereby further reducing the power consumption of the device.

[0087] Based on the foregoing embodiments, various possible implementations of triggering the front camera to capture the first image are described below.

[0088] In a possible implementation, if it is detected that the user has turned on the crowd recognition function, the front camera of the mobile phone is triggered to capture the first image.

[0089] In a possible implementation, if it is detected that the user has turned on the crowd recognition function, the screen state of the mobile phone is detected, and if the screen state of the mobile phone is in a bright screen state, the front camera of the mobile phone is triggered to capture the first image. The interface displayed in the bright screen state of the mobile phone includes, for example, a lock screen interface, a main interface, a third-party application interface, etc.

[0090] In a possible implementation, if it is detected that the user has turned on the crowd recognition function, it is detected whether the mobile phone is unlocked, and if the mobile phone is unlocked, the front camera of the mobile phone is triggered to capture the first image.

[0091] In a possible implementation, if it is detected that the user has turned on the crowd recognition function, it is detected whether the light signal emitted by the proximity light sensor of the mobile phone is blocked, and if it is determined that the light signal emitted by the proximity light sensor is not blocked, the front camera of the mobile phone is triggered to capture the first image. As an example, if the time difference between the light signal emitted by the proximity light sensor of the mobile phone and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the proximity light sensor does not receive the reflected signal, it can be determined that the light signal emitted by the proximity light sensor is not blocked.

[0092] It is understandable that if the user makes or receives a call through the handset, or when the phone is in a bag or pocket, the light signal emitted by the phone's proximity light sensor will be blocked, and the image captured by the front camera will not be able to detect the person's face. At this time, the camera can be stopped from continuously capturing images to reduce device power consumption.

[0093] In a possible implementation, if it is detected that the user has turned on the crowd recognition function, it is detected whether the detection data of the ambient light sensor of the mobile phone is greater than the third threshold value. If it is determined that the detection data of the ambient light sensor is greater than the third threshold value, the mobile phone camera is triggered to capture the first image. Among them, the detection data mainly refers to the ambient light brightness. It should be understood that if the detection data of the ambient light sensor of the mobile phone is greater than the third threshold value, it means that the electronic device is not in a dark environment. The dark environment includes, for example, the mobile phone is in a pocket, or it is currently at night.

[0094] In a possible implementation, if it is detected that the user has turned on the crowd recognition function, the screen orientation of the mobile phone is detected, and if the screen of the mobile phone is facing a preset direction, the front camera of the mobile phone is triggered to capture the first image. In this implementation, the preset direction can be understood as the direction in which the user uses the mobile phone, usually the direction in which the mobile phone screen faces the user, and the direction can be determined by detecting the posture data of the mobile phone, where the posture data includes the pitch angle, the yaw angle, and the roll angle.

[0095] In a possible implementation, if it is detected that the user has turned on the crowd recognition function, it is detected whether the preset application of the mobile phone has been turned on, and if it is determined that the preset application of the mobile phone has been turned on, the front camera of the mobile phone is triggered to capture the first image. In the embodiment of the present application, the preset application includes, for example, a video application, a game application, etc., which is not limited in the embodiment of the present application.

[0096] In one example, if the user has turned on the crowd recognition function and it is detected that the user is using a preset video application, the front camera of the mobile phone is triggered to continuously capture images to detect whether the current user is a user who needs to limit the usage time of the phone.

[0097] In one example, if the user has turned on the crowd recognition function and it is detected that the user is using a preset game application, the front camera of the mobile phone is triggered to continuously capture images to detect whether the current user is a user who needs to limit the usage time of the phone.

[0098] It should be noted that in the above examples, the users whose phone usage time is limited may be children, teenagers or the elderly, and this embodiment of the present application does not limit this.

[0099] In a possible implementation, if it is detected that the user has turned on the crowd recognition function, and it is determined that at least two of the following conditions are met: the screen of the mobile phone is in a bright state, the mobile phone is unlocked, the light signal emitted by the proximity light sensor of the mobile phone is not blocked, the screen of the mobile phone is facing a preset direction, and the preset application of the mobile phone is turned on, then the front camera of the mobile phone is triggered to capture the first image.

[0100] Based on any of the above implementations, the mobile phone camera can be triggered to run in the first mode. Face detection is performed by acquiring a first image with a lower resolution captured by the camera in the first mode. If a person's face is detected in the first image, the mobile phone camera can be triggered to run in the second mode, and a second image with a higher resolution captured by the camera in the second mode can be acquired. By identifying the crowd category of the person in the second image, a preset operation corresponding to the crowd category is executed. The crowd recognition method shown in the above embodiment realizes the function of the mobile phone intelligently sensing the crowd. When the user is identified as a target crowd, the preset operation corresponding to the target crowd is executed to improve the user's experience of using the phone.

[0101] Based on the above embodiments, it can be known that the preset operation includes at least one of the following:

[0102] Adjust the volume, including turning it up, setting it to the maximum value, and turning it down to a preset threshold.

[0103] Adjust the brightness, including increasing the brightness, decreasing the brightness, and adjusting the brightness to the preset brightness.

[0104] The first information is sent, and the first information is used to prompt whether to change the font size, adjust the volume or brightness. The first information can be sent in the form of a pop-up window to facilitate the user to customize the font, volume or brightness.

[0105] Send the second information, the second information is used to prompt the installation (application) risk or traffic risk, and the second information can be sent in the form of a pop-up window.

[0106] The third information is sent to the first external device, and the third information is used for payment verification. The first external device may be a device of a user associated with the current user, for example, if the current user is an elderly person, the associated user is his / her child, or if the current user is a teenager, the associated user is his / her parent; the first external device may also be a server of a payment platform, for example, after the first external device receives the third information, the first external device may send a payment verification message to the user associated with the account.

[0107] Send a fourth message, the fourth message is used to remind you to take a rest. For example, when it is detected that the machine usage time is longer than the preset time, the fourth message is sent. The fourth message can be sent in the form of a pop-up window.

[0108] Sending fifth information to the second external device, where the fifth information is used to indicate the category of people currently using the electronic device, so that the second external device pushes content corresponding to the category of people to the electronic device, and the second external device may be an application server (such as a game server or a video server, etc.);

[0109] Authentication, including face recognition of the owner.

[0110] By setting the above preset operations, relevant restrictions or parameter settings are implemented for different user groups when using mobile phones. The restrictions include restrictions on content, payment, and phone usage time, and the parameter settings include settings for fonts, brightness, volume, and other parameters. The above preset operations can be user-defined or terminal preset, and this embodiment of the application does not impose any restrictions on this.

[0111] In some embodiments, if the mobile phone is a folding screen mobile phone, the folding screen mobile phone includes an inner screen and an outer screen, the inner screen is correspondingly provided with a first camera, and the outer screen is correspondingly provided with a second camera. For example, the first camera is Fig.12 Camera 3 in the image, the second camera is Fig.12 Camera 1 in the.

[0112] As an example, if it is detected that the external screen of the mobile phone is in a bright screen state and the mobile phone is in a folded state, the second camera is controlled to operate in the first mode. As an example, if it is detected that the external screen of the mobile phone is in a bright screen state and the mobile phone is in an unfolded state, the second camera is controlled to operate in the first mode. Based on these two examples, in some embodiments, before controlling the second camera to operate in the first mode, it also includes: detecting that the state of the mobile phone satisfies at least one of the following: the mobile phone is unlocked; the time difference between the light signal emitted by the proximity light sensor (on the external screen) of the mobile phone and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the mobile phone is greater than the third threshold; the external screen of the mobile phone is facing a preset direction; the preset application of the mobile phone is turned on.

[0113] As an example, if it is detected that the inner screen of the mobile phone is in the light-on state and the mobile phone is in the unfolded state, the first camera is controlled to operate in the first mode. Before controlling the first camera to operate in the first mode, it also includes: detecting that the state of the mobile phone satisfies at least one of the following: the mobile phone is unlocked; the time difference between the light signal emitted by the proximity light sensor (on the inner screen) of the mobile phone and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the mobile phone is greater than the third threshold; the inner screen of the mobile phone is facing a preset direction; the preset application of the mobile phone is turned on.

[0114] Based on the above examples, before controlling the first camera to operate in the first mode, or controlling the second camera to operate in the first mode, it also includes: detecting that the state of the electronic device satisfies the second condition; the second condition includes at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the inner screen or outer screen of the electronic device is facing a preset direction; the preset application of the electronic device is turned on.

[0115] As an example, when the second camera operates in the first mode, if it is detected that the electronic device changes from a folded state to an unfolded state, the electronic device controls the first camera to operate in the first mode and turns off the second camera.

[0116] As an example, when the first camera is operating in the first mode, if it is detected that the electronic device changes from an unfolded state to a folded state, the electronic device controls the first camera to turn off and controls the second camera to operate in the first mode; or, the electronic device controls the first camera to turn off.

[0117] The following is combined with Fig.12 A detailed description is given of the crowd recognition method for a foldable screen mobile phone.

[0118] For example, Fig.12 This is a schematic diagram of the structure of a folding screen mobile phone provided in an embodiment of the present application. Fig.12 As shown, the screen of the folding screen mobile phone includes a first screen, a second screen and a third screen. The first screen is the outer screen of the folding screen mobile phone, and the second screen and the third screen are the inner screens of the folding screen mobile phone. The folding screen includes the second screen and the third screen. The folding screen is arranged according to Fig.12 The folding edge shown in (4) is folded to form a second screen and a third screen. The virtual axis where the folding screen is located is a common axis. Among them, the inner screen refers to the screen located inside when the folding screen is in a folded state, and the outer screen refers to the screen located outside when the folding screen is in a closed state. The angle β between the second screen and the third screen is the hinge angle of the folding screen mobile phone. Determining the hinge angle can determine the physical state of the folding screen. The physical state includes the following: Fig.12 The folded state shown in (3) Fig.12 The expanded state shown in (4) or Fig.12 The state of the bracket shown in (2). Fig.12 The folding screen mobile phone shown includes three groups of cameras, which are respectively recorded as camera 1, camera 2 and camera 3. Fig.12 As shown in (1), camera 1 is set in the middle position of the upper part of the first screen, and camera 2 is set on the back panel. Fig.12As shown in (4), camera 3 is set in the middle position of the upper part of the third screen. When the folding screen mobile phone is in the folded state, camera 1 can be regarded as the front camera, and camera 2 can be regarded as the rear camera. Fig.12 As shown in (3) in the middle. When the foldable screen mobile phone is unfolded, camera 3 can be regarded as the front camera, and cameras 1 and 2 can be regarded as the rear cameras.

[0119] Below Fig.12 Taking the folding screen mobile phone shown as an example, examples are given to illustrate under what circumstances the camera of the folding screen mobile phone is turned on and which cameras are turned on.

[0120] In a possible implementation, if the foldable screen mobile phone has the crowd recognition function turned on, when it is detected that the mobile phone is in a folded state and the outer screen (such as Fig.12 The first screen in the figure is in the bright screen state, triggering the camera 1 of the folding screen mobile phone to continuously capture the first image to detect whether there is a human face within the camera range.

[0121] In one possible implementation, if the foldable screen mobile phone has turned on the crowd recognition function, when it is detected that the mobile phone is in a folded state and the external screen is in a bright screen state, and at least one of the above-mentioned second conditions is met, the camera 1 of the foldable screen mobile phone is triggered to continuously capture the first image to detect whether there is a person's face in the camera range.

[0122] In a possible implementation, if the foldable screen mobile phone has the crowd recognition function turned on, when it is detected that the mobile phone is in the unfolded state and the inner screen (such as Fig.12 The second screen and the third screen in the figure are in the bright screen state, triggering the camera 3 of the folding screen mobile phone to continuously capture the first image to detect whether there is a human face within the camera range.

[0123] In one possible implementation, if the foldable screen mobile phone has turned on the crowd recognition function, when it is detected that the mobile phone is in the unfolded state, the inner screen is in the bright screen state, and at least one of the above-mentioned second conditions is met, the camera 3 of the foldable screen mobile phone is triggered to continuously capture the first image to detect whether there is a person's face in the camera range.

[0124] In a possible implementation, if the foldable screen mobile phone meets the conditions for turning on the camera, the current mobile phone is in a folded state and camera 1 is turned on. When it is detected that the mobile phone changes from the folded state to the unfolded state and other conditions for turning on the camera remain unchanged, camera 1 can be turned off and camera 3 can be turned on to detect whether there is a human face within the range of camera 3.

[0125] In a possible implementation, if the foldable screen mobile phone has met the conditions for turning on the camera, the current mobile phone is in the unfolded state, and the camera 3 is turned on. When it is detected that the mobile phone is switched from the unfolded state to the folded state, and other conditions for turning on the camera remain unchanged, the camera 3 can be turned off and the camera 1 can be turned on to detect whether there is a human face within the range of the camera 1.

[0126] It should be noted that the crowd recognition method implemented by other forms of foldable screen mobile phones can refer to Fig.12 The folding screen mobile phone shown has similar implementation principles and technical effects, and the embodiments of the present application do not impose any restrictions on the structural style of the folding screen mobile phone.

[0127] In order to better understand the embodiments of the present application, the structure of the electronic device of the embodiments of the present application is introduced below. Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0128] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a display processing unit (DPU), and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0129] In some embodiments, the electronic device 100 may also include one or more processors 110 .

[0130] In some embodiments, processor 110 may include one or more interfaces.

[0131] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0132] The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, and battery health status. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0133] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor and the baseband processor. The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The wireless communication module 160 can provide solutions for wireless communications including wireless local area networks (WLAN), Bluetooth, global navigation satellite system (GNSS), frequency modulation (FM), NFC, infrared technology (IR), etc. applied to the electronic device 100.

[0134] The electronic device 100 can realize the display function through the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute instructions to generate or change display information.

[0135] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0136] The electronic device 100 can implement a shooting function through an ISP, one or more cameras 193, a video codec, a GPU, one or more display screens 194, and an application processor.

[0137] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0138] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, data files such as music, photos, and videos are stored in the external memory card.

[0139] The internal memory 121 may be used to store one or more computer programs, which include instructions. The processor 110 may execute the instructions stored in the internal memory 121, thereby enabling the electronic device 100 to perform various functional applications and data processing.

[0140] The sensor 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

[0141] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for shooting anti-shake. The gyro sensor 180B can also be used for navigation, somatosensory game scenes, etc.

[0142] The magnetic sensor 180D is used to detect the magnetic field strength of the magnet, obtain magnetic data, and detect the physical state of the folding screen of the electronic device 100 through the magnetic data. The magnet is used to generate a magnetic field. In the embodiment of the present application, the magnetic sensor 180D can be set, for example, Fig.12 In the body corresponding to the back plate shown in (1), the magnet can be arranged, for example, Fig.12In the body corresponding to the first screen shown in (1), the magnet can allow the magnetic sensor 180D to detect magnetic data. As the opening and closing state of the folding screen changes, the distance between the magnetic sensor 180D and the magnet changes accordingly, and the magnetic field strength of the magnet detected by the magnetic sensor 180D also changes. Then, the smart sensor hub can judge the physical state of the folding screen based on the magnetic data obtained by the magnetic sensor 180D under the magnetic field of the magnet. The physical state includes, for example, an unfolded state, a bracket state, or a folded state (closed state). In some embodiments, the sensor 180 may also include a Hall sensor, which can also be used to detect the magnetic field strength of the magnet and output a high / low level, and the physical state of the folding screen of the electronic device 100 is determined by the high / low level.

[0143] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0144] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0145] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0146] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0147] The above-mentioned electronic devices may also be referred to as terminal devices, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. The electronic devices may be mobile phones with touch screens, smart TVs, wearable devices, tablet computers (Pad), computers with wireless transceiver functions, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technologies and specific device forms used by the electronic devices.

[0148] In the embodiment of the present application, in order to realize the function of intelligently sensing scenes of electronic devices, it is necessary to make hardware and software improvements to the camera and processor of the electronic device. The hardware improvements of the electronic device are first described below.

[0149] For example, Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device may include an improved camera 401 and an improved processor 402 .

[0150] The improved camera 401 refers to adding a control circuit and a working circuit corresponding to the newly added shooting mode in the existing camera module to achieve a low-power configuration. For example, the shooting mode of the existing camera module is mode 1, and the camera module includes a working circuit corresponding to mode 1. If the shooting mode adds mode 2, correspondingly, it also involves switching between mode 1 and mode 2. In this regard, the improved camera module includes not only the working circuit corresponding to mode 1, but also the working circuit corresponding to the newly added mode 2, and the control circuit corresponding to the switching between the two modes. It should be understood that more than two shooting modes can be set according to actual application requirements, and the embodiments of the present application do not impose any restrictions on this.

[0151] As an example, the improved camera 401 includes two working modes: a first mode and a second mode, the first mode can be called a low-power shooting mode, and the second mode can be called a normal shooting mode. The resolution of the image captured by the camera 401 in the first mode is lower than the resolution of the image captured in the second mode, and the frame rate of the image captured by the camera 401 in the first mode is lower than the frame rate of the image captured in the second mode. The camera 401 can switch between the two modes.

[0152] Exemplarily, if the conditions for turning on the camera are met, the camera 401 works in the first mode and can perform permanent scanning. The camera 401 continuously captures a first image with a first resolution at a first frame rate to detect whether there is a person within the range of the camera 401, such as detecting whether the first image contains a person's face; if a person is detected, such as detecting that the first image contains a person's face, the camera 401 switches from the first mode to the second mode, and continuously captures a second image with a second resolution at a second frame rate to identify the person's crowd category. The first frame rate is lower than the second frame rate, and the first resolution is lower than the second resolution.

[0153] Based on the above example, camera 401 can dynamically adjust the frame rate and resolution of the captured image to adapt to different needs. For example, camera 401 captures images at a lower frame rate and resolution to detect whether there is a human face within the range of camera 401, and camera 402 captures images at a higher frame rate and resolution to identify the crowd category of the person in the image. Exemplarily, the minimum frame rate of camera 401 can be 1fps, and the maximum frame rate can be 30fps. The minimum resolution of camera 401 can be 120×180, and the maximum resolution can be 480×640. In some embodiments, the maximum frame rate can also be 240fps, and the maximum resolution can also be 2736×3648.

[0154] It should be noted that the embodiments of the present application do not specifically limit the frame rate range and resolution range of the camera to collect images, that is, the minimum frame rate and maximum frame rate of the camera to collect images are not limited, nor are the minimum resolution and maximum resolution of the camera to collect images. In practical applications, the frame rate range and resolution range can be reasonably set according to requirements.

[0155] It should be noted that the camera 401 may be a front camera of the electronic device or a rear camera of the electronic device, which is not limited in this embodiment.

[0156] The improved processor 402 may be a system on chip (SoC). In the embodiment of the present application, when the electronic device triggers the camera 401 to perform a permanent scan to obtain an image of the current scene, the camera 401 may send the image to the SoC for image analysis to detect whether there is a human face, a human crowd category, etc. in the image.

[0157] In order to achieve low power consumption, SoC can support low power consumption AON ISP (Always On ISP), refer to the attached Figure 5 , the camera 401 transmits the image to the AON ISP, which does not process any image effects except for format conversion, and then stores the format-converted image in the on-chip static random access memory (on-chip SRAM). The SoC can also support an extremely low-power core, where computing, algorithm execution, and image storage all work in low-power mode. In addition, the SoC can also support a low-power embedded neural network processor eNPU (emdedded NPU).

[0158] The SoC involved in the embodiments of the present application is described in detail below.

[0159] For example, Figure 5 This is a schematic diagram of the structure of a SoC provided in an embodiment of the present application. Figure 5 As shown, the SoC includes a first processing unit and a second processing unit. The first processing unit includes an image signal processing ISP, a neural network processor NPU and a central processing unit CPU, and the second processing unit includes an I2C bus interface, an AON ISP, an on-chip SRAM, a digital signal processor DSP and an eNPU. In the Soc, the power consumption of the second processing unit is lower than that of the first processing unit. Specifically, the power consumption of the eNPU in the second processing unit is lower than that of the NPU in the first processing unit, and the power consumption of the AON ISP in the second processing unit is lower than that of the ISP in the first processing unit.

[0160] As an example, the first processing unit can be used to process the second image of the second resolution captured by the camera 401. Exemplarily, in the second mode, the camera 401 captures the second image of the second resolution, and after IPS processing, the NPU detects the processed second image of the second resolution, such as detecting the crowd category of the person. Before the first processing unit sends the data (such as image data) to the memory, it can perform security processing (such as encryption processing) and store the data after security processing in a security buffer of the memory. Security processing is used to protect the user's privacy data.

[0161] As an example, the second processing unit can be used to process the above-mentioned first image of the first resolution captured by the camera 401. Exemplarily, in the first mode, the camera 401 captures the first image of the first resolution, and the AON ISP obtains the first image of the first resolution through the I2C bus interface. After being processed by the AON ISP, the eNPU detects the processed first image of the first resolution, for example, detecting whether the first image contains a person's face. The on-chip SRAM in the second processing unit can be used to store the processed first image of the first resolution, and the DSP can be used to notify the eNPU to perform image detection, receive the detection results reported by the eNPU, and report the detection results to the upper-level application. The second processing unit adopts a low-power configuration to reduce the power consumption of the electronic device.

[0162] It should be noted that the embodiment of the present application does not limit the format of the image data transmitted between the first processing unit or the second processing unit and the camera. Exemplarily, the image data can be camera serial interface (Camera Serial Interface, CSI) mobile industry processor interface (Mobile Industry Processor Interface, MIPI) data.

[0163] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system as an example of a layered architecture software system to illustrate the software structure of the electronic device. Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The layered architecture divides the software system of the electronic device into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces.

[0164] Reference Figure 6 The electronic device of the embodiment of the present application includes an application layer (Applications), an application framework layer (Application Framework), a hardware abstraction layer (HAL), a kernel layer (Kernel), a sensor control center (Sensorhub) and a hardware layer.

[0165] The application layer may include a series of applications, and the application layer runs the applications by calling the application programming interface (API) provided by the application framework layer.

[0166] In an embodiment of the present application, the application layer may include a crowd recognition application and a perception module. The crowd recognition application is connected to the perception module, and the crowd recognition application is registered in the perception module. The perception module performs status management and data transmission. For example, when the perception module learns from the second processing module in the Sensorhub that there is a person's face within the camera range, the perception module notifies the first processing module in the HAL so that the first processing module can identify the crowd category of the person, and finally the perception module reports the recognition result to the crowd recognition application.

[0167] In some embodiments, the application layer also includes other applications ( Figure 6 In one possible scenario, multiple applications correspond to the same algorithm. For example, the gaze-on-the-screen application and the gaze-on-AOD application correspond to the gaze detection algorithm. The perception module can be used to uniformly schedule and manage the gaze detection algorithm. In another possible scenario, different applications correspond to different algorithms. For example, the crowd recognition application corresponds to the crowd recognition algorithm (the crowd recognition application also corresponds to the face detection algorithm), and the gaze-on-the-screen application corresponds to the gaze detection algorithm. Both algorithms involve obtaining image data from the underlying camera. The perception module can be used to schedule and manage the priorities among multiple algorithms. There is a possibility that the crowd recognition algorithm and the gaze detection algorithm have the same priority. The perception module can notify the underlying camera to report the image data to the crowd recognition application and the gaze-on-the-screen application at the same time. In this embodiment, the crowd recognition algorithm can be deployed in the first processing module to identify the crowd category of the person. The face detection algorithm can be deployed in the second processing module to detect whether there is a person's face within the camera range. The algorithm shown in this embodiment is only an example.

[0168] In some embodiments, if the electronic device includes a folding screen, the application layer further includes a third processing module, and the third processing module is used to obtain the physical state of the folding screen reported by the second processing module, as well as the state of the internal and external screen cameras (on or off). In addition, the third processing module is also used to notify the first processing module of the state of the internal and external screen cameras.

[0169] In some embodiments, the application may also include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications, which may be system applications or third-party applications, and the embodiments of the present application do not limit this.

[0170] The application framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 6As shown, the application framework layer may include a camera service (CameraService), which is used for priority scheduling and management of all applications that need to use the camera.

[0171] In some embodiments, the application framework layer may also include, for example, a window manager, a content provider, a resource manager, a notification manager, a view system, etc., but the embodiments of the present application do not limit this.

[0172] The hardware abstraction layer may include an AO (always on) service and a first processing module. The AO service may be used to control the on or off of the crowd recognition algorithm in the first processing module, and control the on or off of the face detection algorithm in the second processing module, as well as upper and lower layer data transmission. The first processing module may be used to process images with higher resolution and / or higher frame rate, such as the second image mentioned above, to detect the crowd category of the person in the second image. The first processing module is also used for camera mode switching, for example, the first processing module receives a second instruction from the perception module and controls the camera to switch from the first mode to the second mode.

[0173] The kernel layer is a layer between hardware and software. The kernel layer is used to drive the hardware to make the hardware work. In an embodiment of the present application, the kernel layer may include a camera driver, which is used to drive the camera in the electronic device to work in the first mode or the second mode to collect images with different frame rates and / or resolutions.

[0174] In addition, the core layer may also include display driver, audio driver, sensor driver, motor driver, etc., which are not limited in the embodiments of the present application. Among them, the sensor driver may drive, for example, a proximity light sensor to emit a light signal to detect whether the user holds the electronic device close to the ear to make a call; it may also drive, for example, a gyroscope sensor to detect the posture data of the electronic device; the sensor driver may also drive, for example, an ambient light sensor to detect the ambient light brightness to detect whether the electronic device is in a dark environment, such as a mobile phone in a pocket, etc.

[0175] Sensorhub is used to realize centralized control of sensors to reduce the CPU load. Sensorhub is equivalent to a microprogrammed control unit (MCU), which can run programs for driving multiple sensors. In other words, Sensorhub can support the ability to mount multiple sensors. It can be placed between the CPU and various sensors as an independent chip, or integrated into the application processor (AP) in the CPU.

[0176] In an embodiment of the present application, the Sensorhub may include a second processing module, which may be used to process images with lower resolution and / or lower frame rate, such as the first image mentioned above, to detect whether there is a human face in the first image. Compared with the first processing module, the second processing module is a low-power processing module, and the second processing module is resident or runs in a low-power form. As an example, when the crowd recognition function is turned on, the second processing module is also used to obtain data reported by various sensors, and determine various states of the electronic device based on various sensor data, such as screen state, unlock state, use state, etc. If the device state meets the first condition, the second processing module may send a first shooting instruction to the camera, instructing the camera to scan the image residently in a low-power shooting mode (such as the first mode) to detect whether there is a human face within the camera range. As an example, for a foldable device, the second processing module may determine whether to send a first shooting instruction to the camera (camera on the inner screen or outer screen) by detecting whether the physical state of the folding screen, the screen state, and the device state meet the second condition. As an example, when the second processing module detects that the physical state of the screen of the foldable screen mobile phone has changed, such as from a folded state to an unfolded state, or from an unfolded state to a folded state, the second processing module can control the camera of the foldable screen mobile phone (such as the camera on the outer screen of the mobile phone, and / or the camera on the inner screen of the mobile phone) to turn on or off.

[0177] The hardware layer may include, for example, cameras, various sensors, and AON ISP.

[0178] Understandably, Figure 6 The layers in the hierarchical structure shown and the modules or components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in this application. Figure 6 The modules included in each layer shown are modules involved in the embodiments of the present application, and the modules included in each layer do not constitute a limitation on the structure of the electronic device and the level of module deployment (illustration). Figure 6 The modules shown in can be deployed individually, or several modules can be deployed together. Figure 6 The division of modules in is an example. In some embodiments, Figure 6 The names of the modules shown in are examples.

[0179] Based on the structure of the electronic device shown above, the crowd identification method provided by the embodiment of the present application is described below in combination with specific embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0180] Figure 7 The following is a flow chart of a crowd identification method provided in an embodiment of the present application. Figure 7 As shown, the crowd identification method provided in this embodiment includes:

[0181] Step 701: The target application registers a crowd recognition function in the perception module.

[0182] In this embodiment, the target application is a crowd identification application at the application layer. In one embodiment, the target application obtains information about the crowd identification function from a server (or cloud). After obtaining the information about the crowd identification function, the target application can register the crowd identification function in the perception module so that the perception module can run resident and execute matters related to crowd identification. The information about the crowd identification function includes code information for implementing the crowd identification function, and also includes preset operations corresponding to different crowd categories.

[0183] Step 702: The perception module determines whether to start the crowd recognition function.

[0184] If the perception module determines to start the crowd recognition function, step 703 is executed.

[0185] For example, refer to Figure 2 ,If the user chooses to turn on the crowd recognition function in the ,settings interface of the system application or the third-party application, the system application or the ,third-party application sends a notification to the perception module to ,inform the perception module that the user has turned on the crowd recognition ,function.

[0186] Step 703: The perception module sends a first instruction to the second processing module.

[0187] The first indication is used to instruct the second processing module to detect whether there is a recognizable code within the camera range of the electronic device. The perception module sends the first indication to control the startup of the second processing module, including module power-on, work scene delivery, resource preparation, etc.

[0188] Step 704: The second processing module sends a first shooting instruction to the camera in response to the first instruction, where the first shooting instruction is used to instruct the camera to work in the first mode.

[0189] The second processing module sends a first shooting instruction to control the camera to start, including camera power-on, mode switching, image output resolution and frame rate settings, etc. The camera continuously captures images at a lower frame rate and / or resolution in the first mode, namely, the first image.

[0190] Step 705: The camera sends the first image to the second processing module.

[0191] In response to the first shooting instruction, the camera captures a first image of a first resolution at a first frame rate in a first mode. For example, the first frame rate may be set to 5 fps, and the first resolution may be set to 120×180 or 640×480.

[0192] Step 706: The second processing module identifies whether the first image contains a human face.

[0193] If the second processing module recognizes a human face in the first image, step 707 is executed. Otherwise, the second processing module continues to execute step 706 unless the camera is controlled to be turned off.

[0194] In one embodiment, the second processing module is preset with a facial recognition model, and the facial recognition model can be trained using a lightweight neural network model and is used to identify whether an image contains a human face.

[0195] In some embodiments, the facial recognition model can be deployed on the eNPU of the second processing module, with good real-time performance.

[0196] As an example, the training process for a facial recognition model includes:

[0197] Step a: construct a training set and a test set for the facial recognition model. The training set or the test set includes sample images and annotation results corresponding to the sample images. The annotation results are used to indicate whether there is a human face in the sample image. For example, 0 indicates that there is no human face in the image, and 1 indicates that there is a human face in the image. The sample images in the training set and the test set are different.

[0198] Step b: Based on the initial facial recognition model and the training set, the facial recognition model is trained. Specifically, the sample images of the training set are used as the input of the initial facial recognition model, and the annotation results corresponding to the sample images of the training set are used as the output of the initial facial recognition model to train the facial recognition model.

[0199] Step c: based on the facial recognition model trained in step b and the test set, verify the prediction results of the facial recognition model, and stop training the facial recognition model when the model loss function converges.

[0200] It should be noted that the facial recognition model includes the detection of facial key points, including the detection of key points such as eyes, nose, mouth, eyebrows, etc.

[0201] Step 707: The second processing module sends a first message to the perception module, where the first message is used to notify the perception module that there is a human face within the camera range of the electronic device.

[0202] Step 708: The perception module sends a second instruction to the first processing module.

[0203] The second instruction is used to instruct the first processing module to identify the crowd category of the person within the camera range. The second processing module sends the second instruction to control the startup of the first processing module, including module power-on, work scene delivery, resource preparation, etc.

[0204] Step 709: The first processing module sends a second shooting instruction to the camera in response to the second instruction.

[0205] The second shooting instruction is used to instruct the camera to work in the second mode. The first processing module sends the second shooting instruction to control the camera to switch modes, that is, switch from the first mode to the second mode. The camera continuously captures images at a higher frame rate and / or resolution in the second mode, that is, the second image.

[0206] Step 710: The camera sends a second image to the first processing module.

[0207] In response to the second shooting instruction, the camera captures a second image of a second resolution at a second frame rate in a second mode, and sends the second image to the first processing module. In this embodiment, the second image includes multiple images. Exemplarily, the second frame rate can be set to 30fps, and the second resolution can be set to 1920×1080.

[0208] Step 711: The first processing module identifies the crowd category of the person in the second image.

[0209] As an example, the first processing module inputs the second image into a crowd recognition model, and obtains an image recognition result output by the crowd recognition model, where the image recognition result is used to indicate the crowd category of the person in the second image.

[0210] In one embodiment, the first processing module is preset with a crowd recognition model, and the crowd recognition model can be trained by using a lightweight neural network model and is used to recognize the crowd category of the person in the second image.

[0211] In some embodiments, the crowd recognition model can be deployed on the NPU of the first processing module and has good real-time performance.

[0212] As an example, the training process of a crowd recognition model includes:

[0213] Step a: construct a training set and a test set for a crowd recognition model. The training set or the test set includes sample images (including images of people's faces) and annotation results corresponding to the sample images, and the annotation results are used to indicate the crowd category of the people in the sample images. The sample images in the training set and the test set are different.

[0214] Step b: Based on the initial crowd recognition model and the training set, the crowd recognition model is trained. Specifically, the sample images of the training set are used as the input of the initial crowd recognition model, and the annotation results corresponding to the sample images of the training set are used as the output of the initial crowd recognition model to train the crowd recognition model.

[0215] Step c: based on the crowd recognition model trained in step b and the test set, the prediction result of the crowd recognition model is verified, and when the model loss function converges, the training of the crowd recognition model is stopped.

[0216] For example, Figure 8 A schematic diagram of a crowd identification method provided in an embodiment of the present application is shown as follows: Figure 8 As shown, the front camera of the mobile phone captures one or more first images in a first mode, and the first image is recognized by the second processing module of the mobile phone. If it is recognized that the first image contains a person's face, the front camera of the mobile phone captures one or more second images in a second mode, and the crowd recognition model preset in the first processing module of the mobile phone is used to obtain an image recognition result output by the crowd recognition model. The image recognition result is used to indicate the crowd category of the person in the second image, such as the elderly, teenagers or children.

[0217] Step 712: The first processing module sends a second message to the perception module, where the second message is used to indicate the crowd category of the person in the second image. As an example, the second message includes an identifier of the crowd category.

[0218] Step 713: The perception module sends a third indication to the target application, where the third indication is used to indicate the crowd category of the person in the second image. As an example, the third indication includes an identifier of the crowd category.

[0219] Step 714: The target application controls the execution of preset operations corresponding to the crowd category.

[0220] The target application performs a preset operation corresponding to the crowd category in response to the crowd category indicated in the third indication.

[0221] The preset operation includes at least one of the following:

[0222] Adjust the volume;

[0223] Adjust the brightness;

[0224] Sending a first message, wherein the first message is used to prompt whether to change the font size, adjust the volume or the brightness;

[0225] Sending second information, where the second information is used to prompt installation risk or traffic risk;

[0226] Sending third information to the first external device, where the third information is used for payment verification;

[0227] sending a fourth message, wherein the fourth message is used to remind a person to take a rest;

[0228] Sending fifth information to a second external device, where the fifth information is used to indicate a category of people currently using the electronic device, so that the second external device pushes content corresponding to the category of people to the electronic device;

[0229] Authentication.

[0230] Different crowd categories usually correspond to different preset operations, and users can customize the preset operations.

[0231] It should be noted that, in this embodiment, the first processing module may correspond to Figure 5 The first processing unit shown in the figure, the second processing module may correspond to Figure 5 The second processing unit is shown.

[0232] In the above embodiment, if it is detected that the crowd recognition function of the electronic device has been turned on, the perception module can send a first shooting instruction to the camera through the second processing module (low-power processing module), so that the camera captures the first image at a lower frame rate and / or resolution. If the second processing module recognizes that the first image contains a person's face, it can notify the first processing module so that the first processing module sends a second shooting instruction to the camera, so that the camera captures the second image at a higher frame rate and / or resolution. The first processing module identifies the crowd category of the person in the second image, and then performs a preset operation corresponding to the crowd category.

[0233] In the above scheme, after the user turns on the crowd recognition function, the electronic device detects the face of the person with low power consumption. When the crowd category of the person is determined, the preset operation corresponding to the crowd category is executed, such as pushing relevant content, automatically setting system parameters or prompting the user to set system parameters, etc., so as to realize the function of the electronic device to intelligently perceive specific groups of people and improve the user experience. Since the camera and the second processing module are both configured with low power consumption, the power consumption of executing this scheme is extremely low.

[0234] Fig. 9 A flow chart of a crowd identification method provided in an embodiment of the present application. Figure 7 Based on the embodiment shown, Fig. 9 As shown, the crowd identification method provided in this embodiment includes:

[0235] Step 901: The target application registers a crowd recognition function in the perception module.

[0236] Step 902: The perception module determines whether to start the crowd recognition function.

[0237] If the perception module determines to start the crowd recognition function, step 903 is executed.

[0238] Step 903: The perception module sends a first indication to the second processing module, where the first indication is used to instruct the second processing module to detect whether there is a human face within the camera range of the electronic device.

[0239] Step 904: The second processing module determines whether the first condition is met.

[0240] If the second processing module determines that the first condition is met, step 905 is executed. Otherwise, the second processing module continues the low power consumption detection to determine whether the first condition is met.

[0241] In this embodiment, the first condition includes at least one of the following: the screen state of the electronic device is on; the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the screen of the electronic device is facing a preset direction; the preset application of the electronic device is turned on.

[0242] In this embodiment, if the electronic device has enabled the crowd recognition function and meets the first condition, the camera of the electronic device is triggered to continuously capture the first image. By adding the first condition, the electronic device is prevented from continuously capturing the first image when it is not necessary, thereby further reducing the power consumption of the device.

[0243] Step 905: The second processing module sends a first shooting instruction to the camera, where the first shooting instruction is used to instruct the camera to work in the first mode.

[0244] In some embodiments, if the electronic device is a foldable device, the foldable device includes an inner screen and an outer screen, the inner screen is correspondingly provided with a first camera, and the outer screen is correspondingly provided with a second camera.

[0245] In one case, step 904 can be replaced by: the second processing module detects that the inner screen of the electronic device is in a bright screen state, and the electronic device is in an unfolded state. Correspondingly, step 905 can be: the second processing module sends a first shooting instruction to the first camera. In some embodiments, before the second processing module sends the first shooting instruction to the first camera, it also includes: detecting that the state of the electronic device satisfies the second condition, and the second condition includes at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor (on the inner screen) of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the inner screen of the electronic device is facing a preset direction; the preset application of the electronic device is turned on.

[0246] In one case, step 904 can be replaced by: the second processing module detects that the external screen of the electronic device is in a bright screen state, and the electronic device is in a folded state. Correspondingly, step 905 can be: the second processing module sends a first shooting instruction to the second camera. In some embodiments, before the second processing module sends the first shooting instruction to the second camera, it also includes: detecting that the state of the electronic device satisfies the second condition, and the second condition includes at least one of the following: the electronic device is unlocked; the time difference between the light signal emitted by the proximity light sensor (on the external screen) of the electronic device and the reflected signal of the light signal is greater than the first threshold, and / or the signal strength of the reflected signal is less than the second threshold, and / or the receiving light sensor does not receive the reflected signal; the detection data of the ambient light sensor of the electronic device is greater than the third threshold; the external screen of the electronic device is facing a preset direction; the preset application of the electronic device is turned on.

[0247] Step 906: The camera sends the first image to the second processing module.

[0248] Step 907: The second processing module identifies whether the first image contains a human face.

[0249] If the second processing module recognizes a human face in the first image, step 908 is executed;

[0250] If the second processing module does not recognize the face of the person in the first image, the process jumps back to step 904 .

[0251] Step 908: The second processing module sends a first message to the perception module, where the first message is used to notify the perception module that there is a human face within the range of the electronic device camera.

[0252] Step 909: The perception module sends a second instruction to the first processing module, where the second instruction is used to instruct the first processing module to identify the crowd category of people within the camera range.

[0253] Step 910: The first processing module sends a second shooting instruction to the camera in response to the second instruction, where the second shooting instruction is used to instruct the camera to work in the second mode.

[0254] Step 911: The camera sends a second image to the first processing module.

[0255] Step 912: The first processing module identifies the crowd category of the person in the second image.

[0256] Step 913: The first processing module sends a second message to the perception module, where the second message is used to indicate the crowd category of the person in the second image.

[0257] Step 914: The perception module sends a third indication to the target application, where the third indication is used to indicate the crowd category of the person in the second image.

[0258] Step 915: The target application controls the execution of preset operations corresponding to the crowd category.

[0259] In the above embodiment, if the electronic device has turned on the crowd recognition function and meets the above first condition, the perception module can send a first shooting instruction to the camera through the second processing module, so that the camera captures the first image at a lower frame rate and / or resolution. If the second processing module recognizes that the first image contains a person's face, it can notify the first processing module, and the first processing module sends a second shooting instruction to the camera, so that the camera captures the second image at a higher frame rate and / or resolution. The first processing module identifies the crowd category of the person in the second image, and then executes the preset operation corresponding to the crowd category, such as pushing related content, automatically setting system parameters or prompting the user to set system parameters, etc., to realize the function of the device intelligently sensing a specific group of people. On the one hand, since both the camera and the second processing module adopt low-power configuration, the power consumption of executing the above scheme is extremely low. On the other hand, adding the first condition can avoid the electronic device from performing unnecessary human face detection, thereby further reducing the power consumption of the device.

[0260] The present application also provides a crowd recognition method, which is applied to an electronic device with a flexible screen. The following is an example of a folding screen mobile phone. A group of cameras are respectively set on the inner screen and the outer screen of the folding screen mobile phone to collect image data. The crowd recognition method of this embodiment involves the processing logic of the underlying module of the device when the folding screen changes state. Fig.13 Provide explanation.

[0261] For example, Fig.13 The following is a flow chart of a crowd identification method provided in an embodiment of the present application. Fig.13 As shown, the crowd identification method of this embodiment may include the following steps:

[0262] Step 1301: The second processing module obtains sensor data to determine the change in the physical state of the folding screen.

[0263] In this embodiment, the sensor data includes, for example, a magnetic sensor, a Hall sensor, etc., and the sensor data is obtained to determine whether the physical state of the folding screen has changed. The change in the physical state of the folding screen includes from a folded state to an unfolded state, or from an unfolded state to a folded state.

[0264] Step 1302a: The second processing module controls turning on or off the camera on the inner screen based on the change in the physical state of the folding screen.

[0265] Step 1302b: The second processing module controls turning on or off the camera on the external screen based on the change in the physical state of the folding screen.

[0266] In a possible implementation, if the physical state of the folding screen changes from a folded state to an unfolded state, the second processing module may control the camera on the inner screen to be turned on.

[0267] In one possible implementation, the camera on the outer screen is turned on. If the physical state of the folding screen changes from a folded state to an unfolded state, the second processing module can control the camera on the outer screen to be turned off, and at the same time control the camera on the inner screen to be turned on.

[0268] In one possible implementation, the camera on the inner screen is turned on. If the physical state of the folding screen changes from an unfolded state to a folded state, the second processing module can control the camera on the inner screen to be turned off and the camera on the outer screen to be turned on.

[0269] For example, the camera on the external screen can be Fig.12 The camera 1 shown, the camera on the inner screen can be Fig.12 Camera 3 is shown.

[0270] Step 1302c: The second processing module reports the physical state of the folding screen and the state of the internal and external screen cameras to the third processing module.

[0271] It should be noted that this embodiment does not impose any limitation on the execution order of steps 1302a to 1302c.

[0272] Step 1303: The third processing module sends a notification to the first processing module to notify the status of the internal and external screen cameras.

[0273] This embodiment shows the interaction process between various modules inside the foldable screen mobile phone when the physical state of the screen of the mobile phone changes. Through the above interaction, precise control of the low-power continuous scanning function of the camera is achieved, so that when the user uses the foldable screen mobile phone, the mobile phone can automatically identify the crowd category of the person, and then provide targeted services to the user, such as system parameter optimization, accurate information recommendation, etc., to enhance the user's mobile phone experience.

[0274] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the electronic device includes: a camera 1006, a processor 1001, a communication line 1004 and at least one communication interface ( Fig.10 The communication interface 1003 is used as an example for explanation).

[0275] The camera 1006 may be used to capture images with different frame rates and / or resolutions, and the processor 1001 may be used to detect whether there is a human face in the image, and to identify the crowd category of the human face.

[0276] The processor 1001 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application. In some embodiments, the processor includes a first processing module and a second processing module, the power consumption of the first processing module is higher than the power consumption of the second processing module; the second processing module is used to detect whether there is a human face within the camera range; the first processing module is used to identify the crowd category of the human within the camera range.

[0277] Communications link 1004 may include circuitry to transmit information between the above-described components.

[0278] The communication interface 1003 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0279] In some embodiments, the electronic device may further include a memory 1002 .

[0280] The memory 1002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a communication line 1004. The memory may also be integrated with the processor.

[0281] The memory 1002 is used to store computer-executable instructions for executing the solution of the present application, and the execution is controlled by the processor 1001. The processor 1001 is used to execute the computer-executable instructions stored in the memory 1002, thereby realizing the crowd identification method provided in the embodiment of the present application.

[0282] In some embodiments, the electronic device further includes a display screen 1007 , and the display screen 1007 may be a folding screen.

[0283] The computer-executable instructions in the embodiments of the present application may also be referred to as application program codes, which is not specifically limited in the embodiments of the present application.

[0284] As an example, the processor 1001 may include one or more CPUs, such as Fig.10 CPU0 and CPU1 in.

[0285] As an example, an electronic device may include multiple processors, such as Fig.10 1 and 1005. Each of these processors may be a single-CPU processor or a multi-CPU processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0286] Fig.11 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Fig.11As shown, chip 1100 includes one or more (including two) processors 1120 and a communication interface 1130 .

[0287] In some embodiments, the memory 1140 stores the following elements: executable modules or data structures, or a subset of executable modules or data structures, or an extended set of executable modules or data structures.

[0288] In the embodiment of the present application, the memory 1140 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1120. A portion of the memory 1140 may also include a non-volatile random access memory (NVRAM).

[0289] In the embodiment of the present application, the memory 1140, the communication interface 1130 and the memory 1140 are coupled together through the bus system 1110. The bus system 1110 may include a power bus, a control bus and a status signal bus in addition to a data bus. Fig.11 Various buses are labeled as bus system 1110.

[0290] The method described in the above embodiment of the present application can be applied to the processor 1120, or implemented by the processor 1120. The processor 1120 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware or software instructions in the processor 1120. The above processor 1120 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (digital signal processing, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1120 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present invention.

[0291] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product. The computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0292] The present application also provides a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on an electronic device, the electronic device executes the technical solution in the above embodiment, and its implementation principle and technical effect are similar to those of the above related embodiments, which will not be repeated here.

[0293] Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0294] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the technical solution in the above embodiment. The implementation principle and technical effect are similar to those of the above-mentioned related embodiments and will not be repeated here.

[0295] Computer-readable storage media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. Computer-readable storage media may include: compact disk read-only storage CD-ROM, RAM, ROM, EEPROM or other optical disk storage; computer-readable storage media may include magnetic disk storage or other magnetic disk storage devices. Moreover, any connecting line may also be appropriately referred to as a computer-readable storage medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include compact disks (CDs), laser disks, optical disks, digital versatile discs (DVDs), floppy disks and Blu-ray discs, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers.

[0296] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, user facial information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0297] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A crowd identification method, characterized in that: Applied to an electronic device, the electronic device includes a first interface, the first interface includes a plurality of application icons and switch locks corresponding to the plurality of application icons, the switch lock is used to turn on or off a crowd recognition function of the corresponding application; the method includes: In response to a first operation of a switch lock corresponding to a preset application, enabling a crowd recognition function of the preset application; When it is detected that the preset application is started and it is detected that the state of the electronic device meets the first condition, the camera of the electronic device operates in the first mode; The electronic device obtains a first image captured by the camera in the first mode, and detects whether there is a face of a person in the first image; If the electronic device detects a human face in the first image, controlling the camera to switch from the first mode to a second mode; The electronic device acquires a second image captured by the camera in the second mode, and identifies the crowd category of the person in the second image; The electronic device recognizes that the crowd category of the person is a first category, and controls to execute a preset operation corresponding to the first category; The first condition includes at least one of the following: The screen state of the electronic device is a bright screen state; The electronic device is unlocked; The time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the proximity light sensor does not receive the reflected signal; The detection data of the ambient light sensor of the electronic device is greater than a third threshold; The screen of the electronic device faces a preset direction.

2. The method according to claim 1, characterized in that The electronic device is a foldable device, the foldable device includes an inner screen and an outer screen, the inner screen is correspondingly provided with a first camera, and the outer screen is correspondingly provided with a second camera; the camera of the electronic device operates in a first mode, including: Detecting that the outer screen of the electronic device is in a light-on state and the electronic device is in a folded state, controlling the second camera to operate in a first mode; or It is detected that the inner screen of the electronic device is in a bright screen state and the electronic device is in an unfolded state, and the first camera is controlled to operate in a first mode.

3. The method according to claim 2, characterized in that Before controlling the first camera to operate in the first mode, or controlling the second camera to operate in the first mode, the method further includes: It is detected that the state of the electronic device satisfies a second condition; the second condition includes at least one of the following: The electronic device is unlocked; The time difference between the light signal emitted by the proximity light sensor of the electronic device and the reflected signal of the light signal is greater than a first threshold, and / or the signal strength of the reflected signal is less than a second threshold, and / or the receiving light sensor does not receive the reflected signal; The detection data of the ambient light sensor of the electronic device is greater than a third threshold; The inner screen or outer screen of the electronic device faces a preset direction; The default application of the electronic device is turned on.

4. The method according to claim 2 or 3, characterized in that: The method further comprises: When the second camera operates in the first mode, if it is detected that the electronic device changes from a folded state to an unfolded state, the electronic device controls the first camera to operate in the first mode and turns off the second camera; When the first camera operates in the first mode, if it is detected that the electronic device changes from an unfolded state to a folded state, the electronic device controls the first camera to be turned off, and controls the second camera to operate in the first mode.

5. The method according to claim 1, characterized in that: The camera of the electronic device operates in a first mode, including: The sensing module of the electronic device sends a first instruction to the second processing module of the electronic device, where the first instruction is used to instruct the second processing module to detect whether there is a human face within the camera range; The second processing module sends a first shooting instruction to the camera; The camera operates in the first mode in response to the first shooting instruction.

6. The method according to claim 1, characterized in that The electronic device acquires a first image captured by the camera in the first mode, and detects whether there is a face of a person in the first image, including: The second processing module of the electronic device obtains the first image captured by the camera in the first mode, and detects whether there is a human face in the first image.

7. The method according to claim 1, characterized in that If the electronic device detects a human face in the first image, controlling the camera to switch from the first mode to the second mode includes: If the second processing module of the electronic device detects that there is a human face in the first image, the second processing module sends a first message to the perception module of the electronic device, where the first message is used to notify the perception module that there is a human face within the camera range; The perception module sends a second instruction to the first processing module of the electronic device, where the second instruction is used to instruct the first processing module to identify the crowd category of the person in the second image; The first processing module sends a second shooting instruction to the camera in response to the second instruction, where the second shooting instruction is used to instruct the camera to operate in the second mode.

8. The method according to claim 1, characterized in that The electronic device acquires a second image captured by the camera in the second mode, and identifies a crowd category of the person in the second image, including: The first processing module of the electronic device obtains a second image captured by the camera in the second mode, and identifies the crowd category of the person in the second image.

9. The method according to claim 1, characterized in that: If the electronic device recognizes that the crowd category of the person in the second image is the first category, controlling to execute a preset operation corresponding to the first category includes: If the first processing module of the electronic device recognizes that the crowd category of the person in the second image is the first category, the first processing module sends a second message to the perception module of the electronic device, where the second message is used to indicate that the person in the second image is the first category; The perception module sends a third indication to a target application of the electronic device, wherein the third indication is used to indicate that the person belongs to the first category; The target application controls execution of a preset operation corresponding to the first category.

10. The method according to claim 1, characterized in that The second processing module of the electronic device detects the state of the electronic device.

11. The method according to claim 1, characterized in that: The preset operation includes at least one of the following: Adjust the volume; Adjust the brightness; Sending a first message, wherein the first message is used to prompt whether to change the font size, adjust the volume or the brightness; Sending second information, where the second information is used to prompt installation risk or traffic risk; Sending third information to the first external device, where the third information is used for payment verification; sending a fourth message, wherein the fourth message is used to remind a person to take a rest; Authentication.

12. An electronic device, characterized in that: The electronic device comprises: a camera, a memory and a processor; The camera is used to capture images with different frame rates and / or resolutions, and the processor is used to call the computer program in the memory to execute the method according to any one of claims 1 to 11.

13. The electronic device according to claim 12, characterized in that: The processor comprises a first processing module and a second processing module, the power consumption of the first processing module is higher than the power consumption of the second processing module; The second processing module is used to detect whether there is a human face within the camera range; The first processing module is used to identify the crowd category of people within the camera range.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 11.

15. A chip, characterized in that: The chip comprises a processor, and the processor is used to call a computer program in a memory to execute the method according to any one of claims 1 to 11.

Images