Business management method, electronic device and computer readable storage medium
By keeping the front camera normally open in the electronic device and recognizing image data, it provides contactless human-computer interaction function, which solves the problem of difficulty in interaction when the user is far away from the device, and realizes the provision of multiple functions and the normalization of the application. run.
Patent Information
- Application Number
- CN202311037775.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-16
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-08-16
AI Technical Summary
Existing electronic devices are difficult to achieve human-computer interaction when users are far away from the device, and cannot effectively use the front camera to provide contactless human-computer interaction functions.
Electronic devices provide various functions by keeping the front camera open, identifying the faces and gestures in the collected image data, such as smart scanning codes, air-to-air gestures, smart horizontal and vertical screens and gaze-free screen services, through the conflict management module Avoid memory access conflicts and ensure that different applications are running normally.
It realizes the provision of multiple functions without directly contacting the device, improves the user experience, and ensures the normal operation and resource management of the application.
Smart Images

Figure CN117692540B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminals and communication technologies, and in particular to a service management method, an electronic device, and a computer-readable storage medium. Background Art
[0002] Electronic devices usually need to receive the user's operation of clicking the display screen to achieve human-computer interaction. Sometimes the user is far away from the electronic device and it is not convenient to operate the display screen. In order to solve the above problem, some electronic devices use cameras to provide non-contact human-computer interaction methods, thereby providing users with multiple functions without direct contact with the electronic device. Summary of the invention
[0003] The present application provides a service management method, an electronic device, and a computer-readable storage medium, wherein the electronic device can keep the front camera always on, and the electronic device can recognize faces, gestures, etc. in image data collected by the front camera, thereby providing multiple functions for the user.
[0004] In a first aspect, the present application provides a business management method, which is applied to an electronic device, wherein the electronic device includes a first application and a second application, and the electronic device also includes a display screen and a first camera. The method includes: the electronic device turns on a first function, and when the first function is turned on, the first camera of the electronic device is in an on state, the first function is used to obtain image data collected by the first camera, and the operation of the first application and the second application is based on the image data; the electronic device receives an instruction to turn on the first application at a first time point, and the electronic device receives an instruction to turn off the second application at a second time point, the first time point is earlier than the second time point, the first application uses the image data when the display screen is in a first on-off state, and the second application uses the image data when the display screen is in a second on-off state; the electronic device turns on the first application, and when the first application is turned on, the second application is set to a closed state.
[0005] The first application and the second application use image data in different on and off states, and the first application and the second application use different memory to store data or instructions. When the electronic device has already turned on the second application, even if the electronic device first receives an instruction to turn on the first application, it will not immediately turn on the first application. The electronic device will only turn on the first application when it ensures that the second application is turned off, so that the first application and the second application will not be started at the same time, which will cause a conflict in memory access.
[0006] In combination with the first aspect, in some embodiments, the first on-off state is a screen-on state, and the second on-off state is a screen-off state. When the electronic device opens a first application, the method also includes: the electronic device determines that the second application is in an on state at a first time point, and blocks the first application; after the electronic device closes the second application, the electronic device unblocks the first application, and the time point when the second application is closed is later than the second time point.
[0007] In combination with the first aspect, in some embodiments, the first on-off state is the screen-off state, and the second on-off state is the screen-on state. When the electronic device opens the first application, the method also includes: the electronic device determines that the second application is in the on state at a first time point, closes the second application, and the time point of closing the second application is earlier than the second time point.
[0008] Since the application using image data in the on-screen state cannot access the data or instructions stored in the memory of the application using image data in the off-screen state, the application using images in the off-screen state has a higher priority. The electronic device will not receive the instruction to close the application using images in the off-screen state until the application using images in the off-screen state completes the migration of data or instructions from the underlying memory. Therefore, the electronic device will only close the above application after receiving the instruction, which can avoid problems with memory access between the first application and the second application.
[0009] In combination with the first aspect, in some embodiments, the electronic device further includes a power button, and before the electronic device receives an instruction to start the first application at a first time point, the method further includes: the electronic device receives an operation of the user clicking the power button.
[0010] In combination with the first aspect, in some embodiments, the method also includes: the first application subscribes to image data of multiple different specifications to the first function, the operation of the first application is based on image data of a first specification, and the image data of the first specification is image data of the largest specification selected from image data of multiple different specifications.
[0011] In combination with the first aspect, in some embodiments, the method also includes: the second application subscribes to image data of multiple different specifications to the first function, the operation of the second application is based on image data of a second specification, and the image data of the second specification is image data of the largest specification selected from image data of multiple different specifications.
[0012] Since the first function only supports subscription of image data of the same specification, when different services in the first application or the second application need to subscribe to image data of multiple different specifications from the first function, the electronic device can select the image data with the largest specification and subscribe to the first function. This can ensure that multiple services in the first application or the second application can operate normally.
[0013] In conjunction with the first aspect, in some embodiments, the operation of the first application or the second application is based on the face recognition result, the first application subscribes to the face recognition result from the first function, and the first function also outputs the face recognition result based on the image data. The first function can also output the face recognition result based on the image data, which can meet more types of business.
[0014] In combination with the first aspect, in some embodiments, the electronic device also includes a third application, and the operation of the third application is also based on the image data collected by the first camera. The priority of the third application is higher than the priority of the first application and the second application. Before the electronic device starts the first function, the method also includes: the electronic device determines that the first camera is not occupied by the third application.
[0015] That is to say, the electronic device starts the first function only when the first camera is not occupied by the third application, which can ensure that the third application can run normally and avoid the first function occupying the first camera.
[0016] In combination with the first aspect, in some embodiments, after the electronic device activates the first function, the method further includes: the electronic device detects that a third application is activated, and the electronic device stops providing the data collected by the first camera to the first function.
[0017] That is to say, when the electronic device turns on the first function, if it detects that the third application is started, the image data of the first camera can be output to the third application, and the data collected by the first camera can be stopped from being provided to the first function. This can prevent the first function from occupying the resources of the first camera. Since the first function remains in a normally open state, its priority is lower, and the third application is usually started by the electronic device in response to a user operation, and its priority is higher than that of the first function.
[0018] In combination with the first aspect, in some embodiments, the electronic device includes an application layer, a hardware abstraction layer, a first subsystem layer and a hardware layer, the application layer includes a first application and a second application, the hardware layer includes a display screen and a first camera, the hardware abstraction layer includes a conflict management module, the first subsystem layer includes a first module, and when the first function is turned on, image data collected by the first camera is sent to the first module, and the operation of an application and the second application is based on the image data processed by the first module.
[0019] In combination with the first aspect, in some embodiments, the electronic device receives an instruction to start a first application at a first time point, and the electronic device receives an instruction to close a second application at a second time point, specifically including: the conflict management module in the hardware abstraction layer receives an instruction to start the first application at the first time point, and the conflict management module receives an instruction to close the second application at the second time point.
[0020] In combination with the first aspect, in some embodiments, the first on-off state is a screen-on state, and the second on-off state is a screen-off state. When the electronic device turns on the first application, the method also includes: a conflict management module determines that the second application is in an on state at a first time point, and blocks the first application; the conflict management module unblocks the first application after closing the second application, and the time point when the second application is closed is later than the second time point.
[0021] In combination with the first aspect, in some embodiments, the first on-off state is the screen-off state, and the second on-off state is the screen-on state. When the electronic device opens the first application, the method also includes: a conflict management module determines that the second application is in the on state at a first time point, closes the second application, and the time point for closing the second application is earlier than the second time point.
[0022] In combination with the first aspect, in some embodiments, the first subsystem layer also includes a second module, and the method also includes: the first application subscribes to image data of multiple different specifications to the first module, the operation of the first application is based on the image data of the first specification, and the image data of the first specification is the image data of the largest specification selected by the second module from the image data of multiple different specifications.
[0023] In combination with the first aspect, in some embodiments, the method also includes: the second application subscribes to image data of multiple different specifications to the first function, the operation of the second application is based on image data of second specifications, and the image data of the second specification is image data of the largest specification selected by the second module from image data of multiple different specifications.
[0024] In combination with the first aspect, in some embodiments, the operation of the first application or the second application is based on face recognition results, the first application subscribes to the face recognition results to the first module, and the first function also outputs the face recognition results based on image data through the first module.
[0025] In combination with the first aspect, in some embodiments, the application layer also includes a third application, and the operation of the third application is also based on the image data collected by the first camera. The priority of the third application is higher than the priority of the first application and the second application. Before the electronic device starts the first function, the method also includes: the conflict management module determines that the first camera is not occupied by the third application.
[0026] In combination with the first aspect, in some embodiments, after the electronic device activates the first function, the method further includes: the electronic device detects that a third application is activated, and the first module stops acquiring data collected by the first camera.
[0027] In combination with the first aspect, in some embodiments, the services of the first application include: smart code scanning service, air gesture service, smart horizontal and vertical screen service, auxiliary photo service, gaze ringtone reduction service and gaze without turning off the screen service; the services of the second application include: face sensing and screen turning off display service. Among them, the first application is an application that uses image data when the display screen is in the on-screen state, and the second application is an application that uses image data when the display screen is in the off-screen state.
[0028] In combination with the first aspect, in some embodiments, the services of the first application include: face sensing and screen-off display services; the services of the second application include: smart code scanning services, air gesture services, smart horizontal and vertical screen services, auxiliary photo taking services, gaze ringtone reduction services, and gaze non-screen-off services. Among them, the first application is an application that uses image data when the display screen is in the off state, and the second application is an application that uses image data when the display screen is in the on state.
[0029] In combination with the first aspect, in some embodiments, the first camera is a front camera.
[0030] In a second aspect, the present application provides an electronic device, comprising a display screen, a first camera, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the first camera is used to collect image data, the memory stores a computer program, and when the processor executes the above computer program, the electronic device implements any one of the methods described in the above first aspect.
[0031] In a third aspect, the present application provides a computer-readable storage medium, which stores a computer program or computer instructions, and the aforementioned computer program or computer instructions are executed by a processor to implement any method described in the first aspect above.
[0032] In a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed by a processor, the method described in any one of the above-mentioned first aspects will be implemented.
[0033] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a memory, wherein the memory is used to store computer programs or computer instructions, and the processor is used to execute the computer programs or computer instructions stored in the memory, so that the chip executes any method described in the first aspect above.
[0034] The solutions provided in the second to fifth aspects are used to implement or cooperate with the corresponding methods provided in the first aspect, and therefore can achieve the same or corresponding beneficial effects as the corresponding methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 is a schematic diagram of the hardware structure of the electronic device 100 provided in an embodiment of the present application;
[0036] Figure 2A is an architecture diagram of an electronic device 100 provided in an embodiment of the present application;
[0037] Figure 2B This is an architecture diagram of the smart perception application provided in the embodiment of the present application when the air gesture service is enabled;
[0038] Figure 2C This is an architecture diagram of the smart perception application provided in the embodiment of the present application when the AON screen off service is enabled;
[0039] Figure 3A-3M It is a user interaction diagram when some electronic devices switch between the AON screen-on service and the AON screen-off service provided in the embodiments of the present application;
[0040] Figure 4 It is a timing diagram of the conflict management module controlling the opening and closing of the AON service when the electronic device provided by the embodiment of the present application switches from the screen-on state to the screen-off state;
[0041] Figure 5 It is a timing diagram of the conflict management module controlling the opening and closing of the AON service when the electronic device provided by the embodiment of the present application switches from the screen-off state to the screen-on state;
[0042] Figure 6A-6C This is a schematic diagram of user interaction when multiple AON services are concurrently provided in an embodiment of the present application;
[0043] Figure 7 It is a timing diagram of the intelligent perception software framework subscribing to data when multiple AON services are concurrently provided in an embodiment of the present application;
[0044] Figure 8A-8K It is a user interaction diagram of the AON service provided by the embodiment of the present application being enabled when other applications use the front camera;
[0045] Fig. 9 It is a timing diagram of a conflict management module processing an AON service when other applications use a front camera, provided in an embodiment of the present application;
[0046] Fig.10 It is a flowchart of the business management method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.
[0048] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.
[0049] Electronic devices usually need to receive the operation of users clicking on the display screen to achieve human-computer interaction, but in some scenarios, the user is far away from the electronic device, which is not convenient for directly operating the display screen. In order to enable the electronic device to perceive the user's needs and provide the user with multiple functions when the user is far away from the electronic device, the electronic device in the embodiment of the present application provides a camera real-time online (always-on camera, AON) function. When the electronic device turns on the AON function, the electronic device can collect images in real time through the front camera and recognize faces, gestures, etc. in the image. Then the electronic device can provide the user with multiple functions based on the recognized face gestures, etc.
[0050] Among them, the AON function may include AON screen-on service (function) and AON screen-off service (function). The AON screen-on service is provided by the electronic device when the screen is in the on state, and the AON screen-off service is provided by the electronic device when the screen is in the off state.
[0051] AON screen-on services may include but are not limited to: smart code scanning service, air gesture service, smart horizontal and vertical screen service, assisted photo taking service, stare-attention ringtone reduction service, and stare-attention screen-on service.
[0052] Among them, the smart code recognition service: when the electronic device is in the screen-on state, after the electronic device recognizes the QR code from the image captured by the front camera, it can parse the QR code and jump to the corresponding user interface of the QR code without the user clicking the scan entrance to trigger the code scanning. Optionally, the smart code scanning service can also be realized by keeping the rear camera on all the time.
[0053] Air gesture service: The electronic device can recognize user gestures through the front camera, and respond according to the preset strategy corresponding to the gesture to achieve human-computer interaction. Different gestures can correspond to different preset strategies. For example, the electronic device can take a screenshot after recognizing the user's air grasping gesture, and can turn the page of the displayed user interface when the electronic device recognizes the user's air swipe up / down gesture. The above-mentioned gestures and their corresponding preset strategies are only examples and are not limitations of the embodiments of the present application. Optionally, the smart code scanning service can also be implemented by keeping the rear camera always on.
[0054] Smart horizontal and vertical screen service: When the electronic device is in the screen-on state, if the electronic device detects that the display screen of the electronic device is deflected, and the electronic device detects the direction of the user's face through the front camera, the electronic device can automatically rotate the display screen according to the direction of the user's face, so that the display screen always remains positive relative to the direction of the user's face.
[0055] Assisted photo service: When the electronic device is in the screen-on state, if the electronic device detects that the electronic device has turned on the camera application, and the electronic device detects the direction of the user's face through the front camera, the electronic device can automatically rotate the display screen according to the direction of the user's face, so that the display screen always keeps the positive direction relative to the user's face. When displaying the taken photos, the electronic device can also automatically rotate the photos according to the direction of the user's face, so that the photos displayed on the display screen always keep the positive direction relative to the user's face.
[0056] Staring without turning off the screen service: When the screen of an electronic device is on, if the electronic device detects through the front camera that the user is staring at the electronic device, the display screen of the electronic device will not turn off until the user stops staring at the electronic device.
[0057] Attention ringtone reduction service: When the electronic device is in the screen-on state, if the electronic device detects through the front camera that the user is looking at the electronic device, the electronic device will automatically lower the volume of the incoming call ringtone or the volume of the SMS alert tone. For example, when a user receives an incoming call, the user triggers the mobile phone to automatically lower the volume of the incoming call ringtone by looking at the mobile phone display.
[0058] AON screen-off services may include but are not limited to face-sensing screen-off display services.
[0059] Among them, the face-sensing screen-off display service: The face-sensing screen-off display service can also be called the intelligent always-on display (AOD) service. When the electronic device is in the screen-off state, if the electronic device detects facial features through the front camera, the electronic device automatically displays the screen-off pattern. When the electronic device displays the screen-off pattern, some pixels of the display screen are lit to display information such as the clock, date, and notifications.
[0060] The above-mentioned AON screen-on service and AON screen-off service can be regarded as provided by their respective applications. For example, the air gesture service can be provided by the air gesture application.
[0061] It should be noted that the above-mentioned AON screen-on service and AON screen-off service are only illustrative examples. Electronic devices may include more or fewer AON screen-on services and / or AON screen-off services. The embodiments of the present application do not limit the AON services that can be implemented by electronic devices.
[0062] The electronic device 100 provided in an embodiment of the present application is introduced below.
[0063] Figure 1 It is a schematic diagram of the hardware structure of the electronic device 100 provided in an embodiment of the present application.
[0064] The following is a detailed description of the embodiment using the electronic device 100 as an example. It should be understood that the electronic device 100 may have more than Figure 1 More or fewer components may be shown, two or more components may be combined, or there may be a different configuration of components. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.
[0065] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, an acceleration sensor 180B, a distance sensor 180C, a proximity light sensor 180D, a proximity light sensor 180E, a touch sensor 180F, an ambient light sensor 180G, etc.
[0066] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0067] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0068] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system. In an embodiment of the present application, the cache memory may be a three-level structure, which may be referred to as a first-level cache (L1 cache), a second-level cache (L2 cache), and a third-level cache (LLC), respectively, wherein the first-level cache is closest to the core of the processor 110 and has the highest rate, and the first-level cache stores instructions and data separately. The second-level cache and the third-level cache do not distinguish between instructions and data when storing. The third-level cache is usually shared by multiple cores of the processor 110. The third-level cache is farthest from the core of the processor 110.
[0069] The charging management module 140 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.
[0070] The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 .
[0071] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
[0072] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0073] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0074] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.
[0075] The camera 193 is used to capture still images or videos. The camera 193 may include a lens ( Figure 1 Not shown), photosensor and flexible printed circuit board (FPCB) part ( Figure 1(not shown in the figure). The FPCB is responsible for connecting other components of the camera 193, such as the photosensor, to the image signal processor (ISP) 111, for example, transmitting the raw data output by the photosensor to the ISP. When taking a picture, the shutter of the camera 193 is opened, and light enters and shines on the photosensor. The photosensor converts the light signal into an electrical signal, and then further converts the electrical signal into a digital image signal through analog-digital conversion (ADC) to pass it to the ISP for processing. The ISP can output the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In an embodiment of the present application, the camera 193 includes a front camera 193A, wherein the front camera 193A and the display panel can be located on the same side of the electronic device 100. In this way, when the user looks at the user interface displayed on the display panel, the front camera 193A can capture the facial image of the user. The display panel may reserve a small hole for the position of the front camera 193A, so that the front camera 193A can capture images through the reserved small hole to prevent the lens of the front camera 193A from being blocked by the display panel. Optionally, the front camera 193A may also be embedded in the display panel as an under display camera (UDC).
[0076] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0077] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0078] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0079] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0080] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, applications required for at least one function (such as face recognition function, fingerprint recognition function, mobile payment function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as face information template data, fingerprint information template, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.
[0081] In an embodiment of the present application, the internal memory 121 may include a tightly coupled memory (TCM) memory and a double data rate synchronous dynamic random-access memory (DDR) memory. Among them, the TCM memory is contained in the address mapping space of the memory and can be accessed as a fast memory. TCM is used to provide low-latency memory to the processor, and it does not have the unpredictability unique to the cache memory. The electronic device 100 can use the TCM memory to store important routines, such as interrupt handling routines or real-time tasks that are extremely necessary to avoid the uncertainty of the cache memory. In addition, the electronic device 100 can use the TCM memory to save temporary register data, data types whose local attributes are not suitable for cache, and important data structures such as the interrupt stack. When the above data needs to be used, the processor 110 can directly operate the TCM memory.
[0082] DDR memory can transmit data once on the rising edge and once on the falling edge of each clock, that is, it transmits data twice in one clock cycle. Therefore, DDR memory has a higher data transmission rate.
[0083] The processor 110 may store some frequently used instructions or data in the cache memory, wherein the processor 110 may store the most frequently used instructions or data in the primary cache, and relatively infrequently used instructions or data in the secondary cache or the tertiary cache. When the processor 110 needs to use the instruction or data, it may query the primary cache, the secondary cache, and the tertiary cache in sequence. When the processor 110 fills up the available storage space in the cache memory, the data or instructions may be stored in the internal memory 121 (e.g., TCM or DDR).
[0084] The power consumption of DDR memory is greater than that of TCM memory, and the power consumption of TCM memory is greater than that of LLC memory. The capacity of TCM memory is usually smaller than that of DDR memory.
[0085] In an embodiment of the present application, when the electronic device 100 runs the AON screen-off service, the processor 110 can use LLC and TCM memory to store data or instructions needed for the AON screen-off service. Since the power consumption of the TCM memory is relatively low, this can reduce the power consumption of the electronic device 100 in processing data when the screen is off. When the electronic device 100 turns off the AON screen-off service and turns on the AON screen-on service, the LLC will receive the data or instructions called back by the TCM memory, and the LLC can move part of the data or instructions to the DDR memory. When the electronic device turns off the AON screen-on service and turns on the AON screen-off service, the LLC will move the data or instructions stored in the DDR memory back.
[0086] Not limited to LLC, TCM memory and DDR memory, the cache memory or internal memory in the electronic device 100 for processing AON screen-on service and AON screen-off service may also be other types of memory, and the embodiments of the present application do not impose any restrictions on this.
[0087] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.
[0088] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. When a touch operation is applied to the display screen 194, the electronic device 100 detects the touch operation intensity according to the pressure sensor 180A. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A.
[0089] The acceleration sensor 180B can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and applied to applications such as horizontal and vertical screen switching and pedometers. The acceleration sensor 180B can be used to detect whether the display screen of the electronic device is deflected in the smart horizontal and vertical screen business.
[0090] The distance sensor 180C is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser.
[0091] The proximity light sensor 180D may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects.
[0092] The proximity light sensor 180E is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.
[0093] The touch sensor 180F is also called a "touch panel". The touch sensor 180F can be set on the display screen 194, and the touch sensor 180F and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180F is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180F can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.
[0094] The ambient light sensor 180G is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed brightness of the ambient light.
[0095] The buttons 190 include a power button 190A, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100. The power button 190A may be used to switch the display screen between a bright state and a dark state.
[0096] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting or removing the SIM card interface 195. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications.
[0097] Figure 2A It is a structural diagram of the electronic device 100 according to an embodiment of the present application.
[0098] The layered architecture divides software and hardware into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system may include an application layer, an application framework layer, a hardware abstraction layer, a low-power subsystem, and a hardware layer.
[0099] The application layer may include a series of application packages. For example, the application layer may include applications such as smart perception applications, instant messaging applications, and camera applications (applications may be referred to as applications for short). The embodiments of the present application do not limit the applications that may be included in the application layer. Among them, the smart perception application may include multiple applications that implement AON services. For example, the smart perception application may include a smart code scanning application, an air gesture application, a smart horizontal and vertical screen application, a face sensing screen off display application, and the like. The applications included in the smart perception application are used to implement a single business function, for example, the smart code scanning application is used to implement the smart code scanning business.
[0100] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.
[0101] The hardware abstraction layer is an encapsulation of the kernel driver and can provide an interface to the upper layer. The hardware abstraction layer hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system. In the embodiment of the present application, the hardware abstraction layer includes modules such as intelligent perception interface and service, business management, conflict management, capability management, distributor, screen-on business client, and screen-off business client.
[0102] The intelligent perception interface and service module provides an interface program for the upper-layer intelligent perception application to access the lower-layer intelligent perception software framework.
[0103] Among them, the service management module can be used to manage the subscription and output of image data of multiple AON services. When the smart perception application starts the AON service, the service management module can receive a request to start one or more AON services, and then the service management module can call the conflict management module to determine whether the startup of this AON service will cause a business conflict or a physical conflict.
[0104] The conflict management module can determine the display screen state when the electronic device captures the image, and then instruct the service management module to distribute the service to the corresponding client. The conflict management module can also manage the opening or closing of the AON service to avoid conflicts, and the above conflicts may include service conflicts and physical conflicts. Wherein the service conflict refers to the simultaneous opening of the AON screen-lit service and the AON screen-off service, and the physical conflict refers to the hardware device (such as the front camera) used to implement the AON service of the smart perception application being occupied by other applications. In an embodiment of the present application, the conflict management module may include a service conflict management module and a physical conflict management module. When the smart perception application is opening the AON screen-lit service, the service conflict management module may determine whether there is an unclosed AON screen-off service; or when the smart perception application opens the screen-off service, the service conflict management module may determine whether there is an unclosed AON screen-off service. This can avoid conflicts caused by the simultaneous startup of the AON screen-lit service and the AON screen-off service. The physical conflict management module can determine whether the front camera is already occupied when the smart perception application starts the AON service. When the conflict management module determines that a certain AON service can be opened, the result can be returned to the service management module.
[0105] The capability management module is used to manage the capability information of each business supported by the intelligent perception algorithm platform.
[0106] The distributor is used to distribute the service-related data according to the instructions of the service management module. The distributor can distribute the AON screen-on service to the screen-on service client and the AON screen-off service to the screen-off service client based on the instructions of the service management module.
[0107] The screen-on service client and screen-off service client are used to forward the AON screen-on service and AON screen-off service to the corresponding modules. For example, the screen-on service client can forward the screen-on service to the smart perception software framework, and the screen-off service client can forward the screen-off service to the AON camera module.
[0108] The low-power subsystem is used to implement AON services. The low-power subsystem may include API interfaces and sensor interfaces, smart perception software framework, and AON camera modules.
[0109] The intelligent perception software framework may include a decision-making unit, a control unit, an execution unit, a management unit, etc. These units work together to complete low-power algorithm analysis of business-related data, obtain image analysis results, and report the image analysis results to the intelligent perception application.
[0110] The AON camera module can be used to manage the front camera so that the front camera remains in a normally open state. The AON camera module can also include an image output module and a facial recognition module. Among them, the image output module is used to receive the image captured by the front camera and output the image according to the image specifications subscribed by the intelligent perception software framework. The facial recognition module can identify the face in the image output by the image output module, and then output whether there is a face in the image, when there is a face in the image, the coordinates of the face in the image, the coordinates of the person's eyes and other facial data.
[0111] Not limited to Figure 2A The electronic device 100 may further include the modules shown in the architecture diagram. Figure 2A More software modules are shown, and the software modules of the electronic device 100 in the embodiment of the present application are not limited. For example, the application framework layer may also include a camera manager, which may be used to detect and connect to the camera (e.g., the front camera 193A) of the electronic device 100 and provide an interface for the upper layer application to use the camera.
[0112] After the electronic device 100 is started, the smart perception interface and service module start the smart perception service, and the smart perception application is bound to the smart perception service. The electronic device 100 can receive the user's operation to start the AON service. In response to the operation, the smart perception application will query the capability management module for the capability identifier of the AON service being turned on. Then the smart perception application will register with the service management module to turn on the AON service according to the capability identifier of the AON service.
[0113] The following describes the process of using the smart perception application to start the AON screen-on service and the AON screen-off service.
[0114] Figure 2B The architecture diagram of the smart perception application when the air gesture service is turned on is shown. Here, the air gesture service in the AON screen light service is taken as an example to introduce the method provided by the embodiment of the present application for the smart perception application to start the AON screen light service. The method for the smart perception application to start other AON screen light services can refer to the method for the smart perception application to start the air gesture service, which will not be repeated here.
[0115] like Figure 2B As shown, the smart perception application can send an instruction to enable the air gesture service based on the capability identifier of the air gesture service, where the capability identifier of the air gesture service can be queried by the smart perception application through the capability management module.
[0116] After receiving the instruction to start the air gesture service, the service management module can call the conflict management module, and the conflict management module can coordinate the timing of starting the air gesture service to avoid service conflicts or physical conflicts. The method for coordinating the timing of starting the air gesture service by the conflict management module can refer to the introduction of subsequent embodiments, which will not be expanded here. When the conflict management module determines that the air gesture service can be started, the conflict management module can instruct the service management module to send the instruction to start the air gesture service to the screen-lit service client through the distributor.
[0117] The screen light service client can send the instruction to open the air gesture service to the smart perception software framework, and then the management module in the smart perception software framework can subscribe to the image required by the air gesture service from the AON camera module. Among them, the specifications of the images required by the air gesture service at different stages are different. For example, the smart perception software framework can first subscribe to the AON camera module for an image with a resolution of 160*120, where 160 is the number of horizontal pixels in the image and 120 is the number of vertical pixels in the image. The front camera will always collect image data in the normally open state, and the image data collected by the front camera can be in a Bayer arrangement format, and the image data can also be called a RAW image. Then the ISP can process the RAW image to obtain a YUV image or an RGB image. The AON camera module can obtain the image processed by the ISP and process the image specifications into the image specifications subscribed by the smart perception software framework (ie, a resolution of 160*120). Then the AON camera module can call the image back to the smart perception software framework. The execution module in the smart perception software framework can analyze the content of the image with or without hands through the NPU, and whether the hand is in the starting gesture when there is a hand in the image content.
[0118] Exemplarily, a gesture with a fully open palm and the palm facing the front camera can be a starting gesture. When the execution module in the smart perception software framework confirms that there is a hand in the image content and the hand in the image is the starting gesture, the smart perception software framework can determine that the air gesture service is triggered, and then the execution module can subscribe to a clearer image from the AON camera module, and the clearer image can be an image with a resolution of 320*240. In this way, the AON camera module no longer calls back the 160*120 image to the smart perception software framework, but calls back the 320*240 image. The 320*240 image can be obtained by downsampling the image obtained by the ISP by the image output module in the AON camera module. The smart perception software framework can analyze the subsequent gestures in the image based on the above-mentioned 320*240 resolution image. Taking the air screenshot service in the air gesture service as an example, the subsequent gesture of the air screenshot service can be a hand grasping gesture. The air screenshot service is triggered when the execution module of the smart perception software framework recognizes that the subsequent gesture of the hand in the image is a grasping gesture. The smart perception software framework can call back the message that the remote screenshot service is triggered to the smart perception application, and then the smart perception application can capture the user interface being displayed on the display and save the user interface in the form of a picture to the gallery.
[0119] The stage in which the above-mentioned intelligent perception software framework recognizes whether there is a hand in the content of the image output by the ISP and whether the hand is the starting gesture can be called the first stage of gesture recognition in the air gesture service, and the stage in which the intelligent perception software framework recognizes the subsequent gestures of the hand in the image can be called the second stage of gesture recognition.
[0120] The starting gesture is not limited to the gesture of a fully open palm with the palm facing the front camera. The starting gesture can also be other gestures. The embodiments of the present application do not limit the starting gesture. The above-mentioned air screenshot service and its corresponding subsequent gestures are only examples and are not limitations of the embodiments of the present application. Not limited to the above-mentioned method of identifying gestures, the intelligent perception software framework can also determine whether the air screenshot service is triggered by other methods. For example, the intelligent perception software framework can divide more stages when identifying gestures. The intelligent perception software framework can subscribe to images of different specifications from the AON camera module at each stage of identifying gestures, or the intelligent perception software framework can also continuously identify whether there are preset gestures in multiple frames of images. The embodiments of the present application do not limit the method for the intelligent perception software framework to identify gestures in images.
[0121] Figure 2CThe architecture diagram of the smart perception application when the AON screen off service is turned on is shown. Here, the method of starting the AON screen off service provided by the smart perception application provided in the embodiment of the present application is introduced by taking the perception face screen off display service in the AON screen off service as an example. The method of starting other AON screen off services by the smart perception application can refer to the method of starting the perception face screen off display service by the smart perception application, which will not be repeated here.
[0122] like Figure 2C As shown, the smart perception application can send an instruction to start the face perception screen off display service based on the capability identifier of the face perception screen off display service.
[0123] After receiving the instruction to start the perception face screen off display service, the service management module can call the conflict management module. When the conflict management module determines that the perception face screen off display service can be started, the service management module can be instructed to send the instruction to start the perception face screen off display service to the screen off service client through the distributor, and then the screen off service client can send the instruction to start the perception face screen off display service to the AON camera module. The AON camera module can start the perception face screen off display service, and when the service is triggered, the message is called back to the smart perception application. Specifically, the facial recognition module in the AON camera module can obtain the image processed by the ISP and use an algorithm to identify whether there is a face in the image. When there is a face in the above image, the AON camera module can determine that the face screen off display service is triggered, and then the AON camera module can call back the message that the face screen off display service is triggered to the smart perception application, and then the smart perception application can instruct the display screen to display the screen off pattern.
[0124] The following introduces some application scenarios of AON services of electronic devices provided in embodiments of the present application.
[0125] Figure 3A-3M The user interaction diagram when some electronic devices provided in the embodiments of the present application switch between the AON screen-on service and the AON screen-off service is exemplified.
[0126] Figure 3A The home screen interface 301 displayed by the electronic device 100 is shown as an example. Figure 3A As shown, the main screen interface may include desktop icons of multiple applications, which may include a settings button 302. The settings button 302 is used to jump to the settings interface. For the introduction of the settings interface, please refer to the subsequent embodiments, which will not be expanded here.
[0127] The electronic device 100 may receive an operation of the user clicking the setting button 302, and in response to the operation, the electronic device may display a Figure 3BThe setting interface 310 shown in the figure may include multiple setting options, and the setting options are used to jump to the setting interface of the function of the preset category, and the multiple setting options may include the setting option 311 of the auxiliary function.
[0128] The electronic device 100 may receive an operation of the user clicking on the setting option 311 of the auxiliary function, and in response to the operation, the electronic device may display on the display screen such as Figure 3C The auxiliary function setting interface 320 shown. The auxiliary function setting interface 320 may include multiple auxiliary function setting options, including a smart perception function setting option 321. The right end of the smart perception function setting option 321 may display the word "disabled", indicating that the smart perception function is not currently turned on, that is, the AON screen-on service and the AON screen-off service are not currently started.
[0129] The electronic device may receive an operation of the user clicking on the setting option 321 of the smart sensing function, and in response to the operation, the electronic device may display on the display screen such as Figure 3D The smart perception setting interface 330 shown. The smart perception setting interface 330 may include a sliding switch 331 for the smart perception application, and sliding switches for multiple AON services (for example, a sliding switch for the smart code scanning service, a sliding switch for the smart horizontal and vertical screen service, etc.). The slider of each sliding switch is located on the left side of the sliding switch, which indicates that all AON services are currently in the off state. The sliding switch 331 is used to turn on or off the smart perception application. The current smart perception application is not turned on, and all AON services are also not turned on. The sliding switch for the AON service in the smart perception setting interface 330 may be in a gray state, which means that even if the sliding switch corresponding to the AON service is operated, the corresponding AON service cannot be turned on.
[0130] The electronic device can receive an operation of a user clicking the slide switch 331 of the smart sensing application. In response to the operation, the electronic device 100 can start the smart sensing application, and the slider in the slide switch 331 of the smart sensing application in the smart sensing setting interface 330 moves to the right of the slide switch 331. At the same time, the slide switches of multiple AON services can cancel the grayed-out state. Figure 3EAs shown, the electronic device 100 can receive the operation of the user clicking the sliding switch 332 corresponding to the air screenshot service, and the electronic device 100 can slide the slider of the sliding switch 332 to its right in response to the operation. Since the display screen of the current electronic device 100 is in the bright screen state, and the air screenshot service is the AON bright screen service, the electronic device 100 can also start the air screenshot service in response to the above operation. The smart perception application in the electronic device 100 can turn on the front camera 193A through the AON camera module and put the front camera 193A in a normally open state. At the same time, the smart perception application can send an instruction to turn on the air screenshot service to the service management module through the smart perception interface. After the conflict management module determines that turning on the air screenshot service will not cause a conflict, it can instruct the service management module to send the instruction to turn on the air screenshot service to the bright screen service client, and then the bright screen service client sends the instruction to the smart perception software framework, and the smart perception software framework subscribes to the AON camera module for the 160*120 resolution image required for the first stage of gesture recognition in the air gesture service.
[0131] like Figure 3F As shown, the electronic device 100 can also receive the operation of the user clicking the sliding switch 333 corresponding to the perceived face screen off display service, and in response to the operation, the electronic device 100 can slide the slider of the sliding switch 333 to its right. Since the face screen off display service is an AON screen off service, it needs to run when the display screen of the electronic device 100 is in the screen off state, so the smart perception application needs to wait until the display screen is in the screen off state before starting the perceived face screen off display service. In other words, the user actually turns on the air screenshot service and the perceived face screen off display service in the smart perception setting interface, which actually means that the options of the air screenshot service and the perceived face screen off display service are selected. The electronic device 100 can store the options selected by the user, and when the display screen state matches the display screen state corresponding to the AON service, the corresponding AON service is started. For example, the electronic device 100 can store the options of the air screenshot service selected by the user and the options of the perceived face screen off display service. Among them, the display screen state corresponding to the air screenshot service is the bright screen state, and the display screen state corresponding to the perceived face screen off display service is the screen off state. In this way, when the display screen is in the on-screen state, the electronic device starts the air screenshot service; when the display screen is in the off-screen state, the electronic device can start the face sensing off-screen display service.
[0132] The front camera 193A can continuously capture images and send the image data to the ISP for processing. The AON camera module can process the image output by the ISP to obtain an image with a resolution of 160*120, and then call the image back to the smart perception software framework. The smart perception software framework recognizes that the image content contains a hand and the hand is a preset starting gesture (for example, Figure 3GAfter the hand is fully opened and the palm is facing the front camera 193A, the smart perception software framework can subscribe to the AON camera module for the 320*240 resolution image required for the second stage of gesture recognition in the air gesture service. In this way, the smart perception software framework can recognize subsequent gestures through higher resolution images.
[0133] like Figure 3H As shown, after the front camera 193A captures the image of the user's grasping gesture, the electronic device 100 can capture the image of the currently displayed smart perception setting interface 330 and store the captured image in the gallery. At the same time, the electronic device 100 can display a preview image 334 of the captured image in the lower left corner of the user interface. Among them, the user's grasping gesture can be a subsequent gesture corresponding to the air screenshot service. When the control module of the smart perception software framework recognizes the user's grasping gesture in the second stage of gesture recognition, it can be determined that the air screenshot service is triggered, and then the smart perception software framework can call back the message that the air screenshot service is triggered to the smart perception application, and then the smart perception application can capture the image of the user interface being displayed on the display screen.
[0134] like Fig. 3I As shown, the electronic device 100 may receive an operation of a user clicking the power button 190A, and then the electronic device 100 may respond to the operation by turning off all pixels on the display screen, and the display screen is converted from a bright screen state to a dark screen state. Figure 3J In the screen-off state shown, the front camera 193A always remains on. When the smart perception application determines that the display screen switches from the screen-on state to the screen-off state, it can send an instruction to close the air gesture service to the lower-level business management module. At the same time, the smart perception application can also send an instruction to turn on the face-perceiving screen-off display service to the lower-level business management module. The business management module can call the conflict management module to coordinate the timing of turning on the face-perceiving screen-off display service and turning off the air gesture service to avoid conflicts caused by the simultaneous activation of the AON screen-on service and the AON screen-off service. The method for the conflict management module to coordinate the AON screen-on service and the AON screen-off service can be referred to the introduction of subsequent embodiments, which will not be expanded here.
[0135] When the conflict management module determines that there is no conflict in starting the face sensing screen off display service, the conflict management module can instruct the AON camera module to start the face sensing screen off display service. Figure 3KAs shown, after the front camera 193A captures the user's facial image, multiple pixels in the display screen can be lit up, and the lit pixels can be used to display information such as date, time, and power. Specifically, after the front camera 193A captures the user's facial image, the AON camera module can obtain the above image and identify whether there is a face therein. When the AON camera module recognizes that there is a face in the image captured by the front camera 193A, the message that the face-sensing screen-off display service is triggered can be called back to the smart perception application, and then the smart perception application can instruct the display screen to display the screen-off display information. It should be noted that when the electronic device displays the screen-off display information, most of the pixels in the display screen are not lit, and the electronic device is still in the screen-off state.
[0136] like Figure 3L As shown, the electronic device 100 receives the operation of the user clicking the power button 190A, and in response to the operation, the electronic device 100 can be converted from the screen-off state to the screen-on state as shown in FIG. Figure 3M In the bright screen state shown, the electronic device 100 can display the smart perception setting interface 330. In addition, the smart perception application can turn off the face screen display service and start the air gesture service in the air screenshot service.
[0137] When the electronic device switches from the screen-on state to the screen-off state, the smart perception application needs to send a command to turn off the AON screen-on service and a command to turn on the AON screen-off service; when the electronic device switches from the screen-off state to the screen-on state, the smart perception application needs to send a command to turn off the AON screen-off service and a command to turn on the AON screen-on service. The above-mentioned AON service that is turned on can be a user's Figure 3E The electronic device 100 will store the AON screen-on service option and the AON screen-off service option enabled by the user in the smart perception setting interface 330, so that when the electronic device 100 is in the screen-on state, the smart perception application can start the AON screen-on service corresponding to the AON screen-on service option enabled by the user in the smart perception setting interface 330, and close the running AON screen-off service; when the electronic device 100 is in the screen-off state, the smart perception application can start the AON screen-off service corresponding to the AON screen-off service option enabled by the user in the smart perception setting interface 330, and close the running AON screen-on service.
[0138] Since the opening and closing of the AON screen-on service and the AON screen-off service are controlled by different modules in the smart perception application, when the smart perception application issues an instruction to open (or close) the AON screen-on service and an instruction to close (or open) the AON screen-off service, it is difficult to determine the timing of the module responsible for the AON screen-on service receiving the instruction to open (or close) the AON screen-on service and the module responsible for the AON screen-off service receiving the instruction to close (or open) the AON screen-off service. If the AON screen-on service is turned on when the AON screen-off service is not closed, or the AON screen-off service is turned on when the AON screen-on service is not closed, the AON screen-off service and the AON screen-on service may run at the same time. The above situation can be called a service conflict. The service conflict management module in the conflict management will coordinate the timing of turning the AON service on or off to avoid service conflicts.
[0139] It should be noted that the reason why the AON screen-off service and the AON screen-on service should be avoided from running at the same time is that the AON screen-off service and the AON screen-on service are handled by different memory domains. The AON screen-off service is mainly stored in the TCM memory, and the AON screen-on service is mainly stored in the DDR memory. When the AON screen-on service is started, the LLC will move the data or instructions to the DDR. When the AON screen-on service is turned off, the data or instructions in the DDR can be moved back to the LLC; when the AON screen-off service is turned off, the data or instructions in the TCM can be called back to the LLC. If the AON screen-off service and the AON screen-on service run at the same time, some data or instructions may be stored in the TCM memory, and the DDR memory reads the data or instructions called back by the TCM memory through the LLC. Some data or instructions in the TCM memory cannot be used directly by the DDR. When the DDR memory reads and uses the above data or instructions stored in the TCM memory, the AON service will be invalid.
[0140] The following specifically introduces how the conflict management module controls the activation of the AON screen-on service and the AON screen-off service.
[0141] Figure 4 This is a timing diagram of the conflict management module controlling the opening and closing of the AON service when the electronic device provided in an embodiment of the present application switches from a screen-on state to a screen-off state.
[0142] like Figure 4 As shown, the method may include but is not limited to:
[0143] S401. The smart perception application receives an operation in which a user selects an A screen-on service option and a B screen-off service option.
[0144] When the electronic device 100 is in the screen-on state, the smart sensing application can receive the user's operation of selecting the A screen-on service option and the B screen-off service option in the smart sensing setting interface. The A screen-on service can be one or more AON screen-on services, and the B screen-off service can be one or more AON screen-off services.
[0145] by Figure 3D-Figure 3F Taking the embodiment shown as an example, the A screen-on service may be, for example, an air-screen capture service in the air gesture service, and the B screen-off service may be, for example, a face-sensing screen-off display service. The electronic device may receive a single-click operation of the user on the sliding switch 332 corresponding to the air-screen capture service and the sliding switch 333 corresponding to the face-sensing screen-off display service, and the single-click operation is the operation of the user selecting the A screen-on service option and the B screen-off service option.
[0146] S402. The smart perception application sends an instruction to the application framework layer to enable the A screen light service.
[0147] S403. The application framework layer sends an instruction to the smart perception interface and service module to start the A screen lighting service.
[0148] S404. The intelligent perception interface and service module sends an instruction to the conflict management module to enable the A screen lighting service.
[0149] The electronic device 100 can store the identifier of the AON service selected by the user, and then turn on the AON service when the display screen is in the state corresponding to the AON service. Since the display screen of the electronic device is in the screen-on state after the user selects the A screen-on service option and the B screen-off service option, the smart perception application can send an instruction to turn on the A screen-on service to the application framework layer. Then the application framework layer can send the instruction to turn on the A screen-on service to the smart perception interface and service module. The smart perception interface and service module can send the instruction to the service management module, and the service management module can call the conflict management module, and the conflict management module will determine whether the A screen-on service can be turned on.
[0150] S405. The business conflict management module in the conflict management module determines that there is no business conflict.
[0151] S406. The conflict management module sends an instruction to enable the A screen lighting service to the smart perception software framework.
[0152] When the conflict management module receives the instruction to start the A screen-on service, it can determine whether there is an AON screen-off service in operation. Since there is currently no AON screen-off service in operation, starting the A screen-on service will not result in the AON screen-on service and the AON screen-off service running at the same time, which means that there is no service conflict. Then the conflict management module can instruct the service management module to issue the instruction to start the A screen-on service to the screen-on service client through the distributor, and then the screen-on service client sends the instruction to start the A screen-on service to the smart perception software framework through the API interface.
[0153] S407. The intelligent perception software framework subscribes to the AON camera module for data required for the A screen lighting service.
[0154] After receiving the instruction to start the A screen bright service, the intelligent perception software framework can subscribe to the data required by the A screen bright service from the AON camera module. The above data may include image data and face data, etc. The above image data can be obtained by the AON camera module downsampling the YUV image or RGB image obtained by the ISP. The face data can be the face coordinates, eye coordinates and other data obtained by the AON camera module after recognizing the face in the above image through the face recognition algorithm.
[0155] Among them, the smart perception software framework can subscribe to the image resolution from the AON camera module. In this way, after the AON camera module receives the YUV image or RGB image processed by the ISP, it can downsample the above image, where the resolution of the downsampled image is the same as the resolution subscribed by the smart perception software framework. The AON camera module can call back the data of the downsampled image to the smart perception software framework, or the AON camera module can perform face recognition based on the downsampled image to obtain face data, and then call back the face data to the smart perception software framework. The smart perception software framework can receive the AON callback data and then determine whether the A screen light service is triggered (refer to Figure 3G , Figure 3H embodiment shown).
[0156] S408: The electronic device receives an operation of the user pressing a power button.
[0157] S409. The smart perception application sends an instruction to the application framework layer to enable the B screen-off service.
[0158] S410. The application framework layer sends an instruction to the intelligent perception interface and service module to start the B screen off service.
[0159] S411. The intelligent perception interface and service module sends an instruction to the conflict management module to start the B screen off service.
[0160] After the electronic device 100 receives the operation of the user pressing the power button, it will switch from the on screen state to the off screen state. When the electronic device switches to the off screen state, the smart perception application will send an instruction to the application framework layer to turn on the B screen off service. The application framework layer sends the instruction to turn on the B screen off service to the service management module through the smart perception interface and the service module. The service management module can call the conflict management module to determine whether turning on the B screen off service will cause a conflict.
[0161] S412: The service conflict management module in the conflict management module determines that a screen-on service is running.
[0162] S413. The conflict management module sends an instruction to close the A screen lighting service to the smart perception software framework.
[0163] S414. The intelligent perception software framework cancels the subscription of the AON camera module to the data required for the A screen lighting service.
[0164] S415: The conflict management module determines that the screen-on service has been closed.
[0165] S416. The conflict management module subscribes to the AON camera module for data required for the B screen off service.
[0166] When the service conflict management module receives an instruction to turn on the B screen-off service, it will first determine whether there is a screen-on service running. Since the current A screen-on service has not been closed, if the B screen-off service is directly turned on, the AON screen-on service and the AON screen-off service may run at the same time. Therefore, the conflict management module can send an instruction to turn off the A screen-on service to the service management module, and then the service management module sends the instruction to the screen-on service client through the distributor. The screen-on service client sends the instruction to turn off the A screen-on service to the smart perception software framework, and the smart perception software framework unsubscribes from the data required for the A screen-on service to the AON camera module. When the conflict management module determines that the A screen-on service is closed, it can instruct the service management module to turn on the B screen-off service. The service management module can send the instruction to turn on the B screen-off service to the screen-off service client through the distributor, and then the screen-off service client can subscribe to the data required for the B screen-off service from the AON camera module.
[0167] refer to Figure 3I-3J In the embodiment shown, when the electronic device switches from the on-screen state to the off-screen state, the smart perception application can issue an instruction to start the face sensing off-screen display service. The conflict management module will first determine whether the air screenshot service is closed. If the air screenshot service has not been closed, the conflict management module will first instruct the service management module to close the air screenshot service, and then start the face sensing off-screen display service.
[0168] S417. The smart perception application sends an instruction to the application framework layer to close the A screen lighting service.
[0169] S418. The application framework layer sends an instruction to close the A screen lighting service to the smart perception interface and service module.
[0170] S419. The intelligent perception interface and service module sends an instruction to the conflict management module to close the A screen lighting service.
[0171] S420: The conflict management module determines that the screen-on service of A has been closed and no instruction needs to be issued.
[0172] When an electronic device switches from a screen-on state to a screen-off state, the data or instructions for the AON screen-on service stored in the DDR will be moved to the LLC. After the move, the smart perception application can instruct the module responsible for the AON screen-on service in the smart perception application to turn off the A screen-on service. Then the module responsible for the AON screen-on service will send an instruction to turn off the A screen-on service. The instruction is sent to the service management module through the application framework layer, and the service management module calls the conflict management module to determine whether to turn off the A screen-on service. Since the A screen-on service has been turned off, the instruction to turn off the A screen-on service does not need to be sent to the screen-on service client.
[0173] It should be noted that Figure 4 The time for the electronic device 100 to execute step S417-step S420 is only an example. In the scenario where the electronic device switches from the screen-on state to the screen-off state, when the module responsible for the AON screen-on service in the smart perception application receives the instruction to turn off the AON screen-on service later than the module responsible for the AON screen-off service receives the instruction to turn on the AON screen-off service, the method for the conflict management module to control the opening or closing of the AON screen-on service and the AON screen-off service can refer to Figure 4 When the module responsible for the AON screen-on service in the smart sensing application receives an instruction to turn off the AON screen-on service earlier than the module responsible for the AON screen-off service receives an instruction to turn on the AON screen-off service, the conflict management module can turn off the AON screen-on service first and then turn on the AON screen-off service according to the instruction sequence.
[0174] Figure 5 This is a timing diagram of the conflict management module controlling the opening and closing of the AON service when the electronic device provided in an embodiment of the present application switches from a screen-off state to a screen-on state.
[0175] like Figure 5 As shown, the method may include but is not limited to:
[0176] S501. The intelligent perception application sends an instruction to the application framework layer to enable the B screen-off service.
[0177] S502. The application framework layer sends an instruction to the smart perception interface and service module to start the B screen-off service.
[0178] S503. The intelligent perception interface and service module sends an instruction to the conflict management module to start the B screen off service.
[0179] S504: The business conflict management module in the conflict management module determines that there is no business conflict.
[0180] S505. The conflict management module subscribes to the AON camera module for data required for the B screen-off service.
[0181] When the electronic device 100 is in the screen-off state, the smart perception application can send an instruction to the application framework layer to start the B screen-off service. The B screen-off service can be started by the user when the electronic device is in the screen-on state. When the electronic device receives the user's operation to start the B screen-off service, it can wait until the electronic device is in the screen-off state and then send the instruction to start the B screen-off service to the application framework layer.
[0182] refer to Figure 3A-Figure 3F In the embodiment shown, the screen-off service B may be a face-sensing screen-off display service. When the electronic device is in the screen-on state, it may receive an operation of the user clicking the corresponding sliding switch 333 for the face-sensing screen-off display, which is used to turn on the face screen-off display service. The electronic device may save the user's settings for turning on the face screen-off display service, and when the electronic device is switched to the screen-off state, it may send an instruction to start the face screen-off display service to the application framework layer.
[0183] After the application framework layer receives the instruction to start the B screen-off service, it can send the instruction to the business management module through the intelligent perception interface and service. The business management module can call the conflict management module to determine whether starting the B screen-off service will cause a business conflict. The conflict management module can determine that there is no AON screen-on service running, which indicates that starting the B screen-off service immediately will not cause a business conflict. Then the conflict management module can instruct the business management module to issue an instruction to start the B screen-off service. The distributor can send the instruction to start the B screen-off service to the screen-off service client, and then the screen-off service client subscribes to the AON camera module for the image or face data it needs.
[0184] S506: The electronic device receives an operation of the user pressing a power button.
[0185] S507. The smart perception application sends an instruction to the application framework layer to enable the A screen lighting service.
[0186] S508. The application framework layer sends an instruction to the smart perception interface and service module to start the A screen lighting service.
[0187] S509. The intelligent perception interface and service module sends an instruction to enable the A screen lighting service to the conflict management module.
[0188] S510. The service conflict management module in the conflict management module determines that the screen-off service is running, and blocks the screen-on service A from starting.
[0189] After the electronic device 100 receives the operation of the user pressing the power button, it can light up the display screen. The electronic device is converted from the screen-off state to the screen-on state. Then the smart perception application can send an instruction to the application framework layer to turn on the A screen-on service. The application framework layer sends an instruction to turn on the A screen-on service to the business management module through the smart perception interface and service. The conflict management module receives the instruction sent by the business management module, determines that there is still a running screen-off service, and then the conflict management module can block the start of the A screen-on service. In other words, the conflict management module may temporarily not instruct the business management module to distribute the A screen-on service.
[0190] S511. The intelligent perception application sends an instruction to the application framework layer to close the B screen-off service.
[0191] S512. The application framework layer sends an instruction to close the B screen-off service to the intelligent perception interface and service module.
[0192] S513. The intelligent perception interface and service module sends an instruction to the conflict management module to close the B screen-off service.
[0193] S514. The service conflict management module in the conflict management module determines that the screen-off service B can be closed.
[0194] S515. The conflict management module cancels the subscription of the data required for the screen-off service of B to the AON camera module.
[0195] After the electronic device 100 is converted to the screen-on state, the data stored in the TCM memory will be called back to the LLC. When the data in the memory is moved, the smart perception application will send an instruction to close the B screen-off service. Then the instruction is sent to the conflict management module through the application framework layer and the smart perception interface and service module. The service management module in the conflict management module can determine that the screen-on service has not been started, and the B screen-off service can be closed. Then the conflict management module can instruct the distributor to distribute the instruction to close the B screen-off service to the screen-off service client, and then the screen-off service client cancels the subscription to the data required for the B screen-off service.
[0196] S516. The conflict management module determines that the screen-off service has been closed and there is a screen-on service to be restored.
[0197] S517. The conflict management module sends an instruction to enable the A screen lighting service to the smart perception software framework.
[0198] S518. The intelligent perception software framework subscribes to the AON camera module for data required for the A screen lighting service.
[0199] The service conflict management module in the conflict management module can open the blocked A screen-on service after the screen-off service is closed. The conflict management module can instruct the service management module to distribute to the screen-on service client through the distributor, and then the screen-on service client instructs the smart perception software framework to open the A screen-on service. In this way, the smart perception software framework can subscribe to the data required by the A screen-on service from the AON camera module, and then determine whether the A screen-on service is triggered through the above data.
[0200] When the electronic device changes from the screen-off state to the screen-on state, the module responsible for the AON screen-off service in the smart perception application receives the instruction to turn off the AON screen-off service later than the module responsible for the AON screen-on service receives the instruction to turn on the AON screen-on service. The method for the conflict management module to control the turning on or off of the AON screen-on service and the AON screen-off service can refer to Figure 5 The method shown.
[0201] Figure 4 The timing diagram shown demonstrates the scenario where the smart perception application starts the AON screen-off service while the AON screen-on service is running. In this scenario, the conflict management module will actively send an instruction to shut down the running AON screen-on service and then continue to start the AON screen-off service. This situation usually occurs when the electronic device changes from the screen-on state to the screen-off state. For the convenience of description, this scenario is called scenario 1.
[0202] Figure 5 The timing diagram shown demonstrates a scenario in which the smart perception application turns on the AON screen-on service while the AON screen-off service is running. In this scenario, the conflict management module will block the startup of the AON screen-on service. The conflict management module will wait for the instruction to shut down the AON screen-off service, and then shut down the running AON screen-off service according to the instruction. When the conflict management module determines that the running AON screen-off service has been shut down, the startup of the blocked AON screen-on service will be restored. This scenario usually occurs when the electronic device is converted from the screen-off state to the screen-on state. For ease of description, this scenario is called scenario two.
[0203] That is to say, in scenario one and scenario two, the conflict management module controls the opening and closing of the AON screen-on service and the AON screen-off service in different ways. In scenario one, the electronic device uses DDR memory when running the AON screen-on service. When the electronic device switches from the screen-on state to the screen-off state, the data or instructions in the DDR memory will be moved to the LLC. After receiving the instruction to start the AON screen-off service, the conflict management module can directly instruct the AON screen-on service to be turned off without waiting for the instructions or data in the DDR to be completely moved to the LLC. In scenario two, when the electronic device runs the AON screen-off service, TCM memory is used. When the electronic device switches from the screen-off state to the screen-on state, when the conflict management module receives an instruction to start the AON screen-on service, it will first block the AON screen-on service, and then start the AON screen-on service after the instructions or data stored in the TCM are called back to the LLC. This can avoid the AON screen-on service from operating the DDR memory, but the data or instructions actually used by the AON screen-on service are still stored in the TCM memory, causing the electronic device to read the data called back from the TCM memory through the LLC in the DDR memory, making the AON service invalid. The electronic device 100 will only execute step S511 after the instructions or data stored in the TCM have completed the callback. In addition, since the number of AON screen-on services is greater than the number of AON screen-off services, the above method is easier to manage.
[0204] Different AON services require different image specifications, and the AON camera module only accepts subscriptions to images of a single specification at a time, as well as facial data recognized based on images of that specification. For example, the Smart Perception Software Framework can subscribe to images of specification A and facial data recognized based on images of specification A from the AON camera module, but the Smart Perception Software Framework cannot simultaneously subscribe to images of specifications A and B from the AON camera module, nor can it simultaneously subscribe to images of specification A and facial data recognized based on images of specification B from the AON camera module.
[0205] In some embodiments, in order to ensure the concurrency of multiple AON services, the management module in the smart perception software framework can manage the image specifications of the AON service subscription.
[0206] The following first introduces a scenario where multiple AON services are concurrently used. Figure 6A-6C The user interaction diagram when multiple AON services are concurrently provided in the embodiments of the present application is exemplified.
[0207] like Fig. 6A As shown, the electronic device 100 displays the smart perception setting interface 330. The slider of the sliding switch 332 of the air screenshot service is at its right end, which can be the electronic device 100 receiving and responding to the following Figure 3EThe user clicks the sliding switch 332, which is used to start the air screenshot service. Since the electronic device 100 is in the screen-on state, the smart perception application can issue an instruction to start the air screenshot service, and then the smart perception software framework will subscribe to the AON camera module for the data required for the air screenshot service.
[0208] The electronic device may receive an operation of the user clicking the slide switch 601 of the gaze-on-screen service, and in response to the operation, the electronic device 100 may start the gaze-on-screen service. Figure 6B As shown, the image captured by the front camera contains gestures and faces. The AON camera module can simultaneously call back the data used to implement the air screenshot service and the staring without turning off the screen service to the intelligent perception software framework to ensure the concurrent execution of the two AON services.
[0209] like Figure 6C As shown, when the front camera captures the grasping gesture of a person, the remote screenshot service is triggered. The electronic device 100 can display a preview image 602 of the intercepted smart perception setting interface 330 in the lower left corner of the user interface.
[0210] The following describes in detail the method of subscribing data by the intelligent perception software framework when multiple AON services are concurrent with each other, combined with the timing diagram.
[0211] Here, we take the air gesture service and the gaze-on-screen service as examples to introduce the method of the intelligent perception software framework subscribing to data when multiple AON services are concurrent. For other AON services or more AON services concurrently, the method of the intelligent perception software framework subscribing to data can refer to the method of the intelligent perception software framework subscribing to data to the AON camera module when the air gesture service and the gaze-on-screen service are concurrent. I will not go into details here.
[0212] like Figure 7 As shown, the method for the intelligent perception software framework to subscribe to data when multiple AON services are concurrent may include but is not limited to:
[0213] S701. The smart perception application receives an operation from a user to start an air gesture service.
[0214] S702. The smart perception application sends an instruction to the application framework layer to enable the air gesture service.
[0215] S703. The application framework layer sends an instruction to enable the air gesture service to the smart perception interface and service module.
[0216] S704. The smart perception interface and service module sends an instruction to enable the air gesture service to the smart perception software framework.
[0217] The smart sensing application can receive the user's operation to start the air gesture service, and the air gesture service can be, for example, the air screenshot service in the air gesture service. Figure 3E Taking the embodiment shown as an example, the operation may be an operation in which the user clicks the sliding switch 332 corresponding to the air screenshot service in the smart perception setting interface 330. In response to the operation, the smart perception application may send an instruction to the application framework layer to enable the air screenshot service. Then, the application framework diagram is sent to the service management module through the smart perception interface and the service module. The service management module instructs the distributor to distribute the instruction to the smart perception software framework.
[0218] S705. The management module in the intelligent perception software framework determines that the gesture recognition stage 1 requires an image of specification A.
[0219] S706. The management module in the intelligent perception software framework subscribes to the A specification image from the AON camera module.
[0220] S707, the AON camera module calls back the A-specification image to the intelligent perception software framework.
[0221] After the smart perception software framework receives the instruction to start the air gesture service, the management module in the smart perception software framework can subscribe to the A specification image required for the first stage of gesture recognition from the AON camera module. The above-mentioned A specification image can be an image with a resolution of 160*120. The AON camera module can downsample the image output by the ISP to 160*120, and then call it back to the smart perception software framework. In the first stage of gesture recognition, the AON camera module can call back the A specification image to the smart perception software framework multiple times.
[0222] S708. The management module in the intelligent perception software framework determines that the second stage of gesture recognition requires an image of specification B.
[0223] S709. The management module in the intelligent perception software framework subscribes to the B-specification image from the AON camera module.
[0224] S710, AON camera module calls back the B-specification image to the intelligent perception software framework.
[0225] When the smart perception software framework recognizes the presence of a starting gesture in the image based on the A-specification image, the second stage of gesture recognition is triggered. The smart perception software framework can subscribe to the AON camera module for the B-specification image required for the second stage of gesture recognition. The AON camera module can process the image output by the ISP into the B-specification and then call the image back to the smart perception software framework. The above-mentioned B-specification image can be, for example, a 320*240 resolution image.
[0226] S711. The smart perception application receives an operation from a user to enable the "watch without turning off the screen" service.
[0227] S712. The smart perception application sends an instruction to the application framework layer to enable the "watch without turning off the screen" service.
[0228] S713. The application framework layer sends an instruction to the smart perception interface and service module to enable the "watch without turning off the screen" service.
[0229] S714. The smart perception interface and service module sends an instruction to the smart perception software framework to enable the "watch without turning off the screen" service.
[0230] Smart sensing applications can receive the user's operation to enable the "attention without turning off the screen" service. Fig. 6A Taking the shown embodiment as an example, the above operation may be an operation in which the user clicks the sliding switch 601 of the gaze-not-turning-off service. In response to this operation, the smart perception application may send an instruction to the application framework layer to enable the gaze-not-turning-off service, and then the smart perception software framework may receive the instruction to enable the gaze-not-turning-off service.
[0231] S715. The management module in the intelligent perception software framework determines that the "watch the screen without turning off" service needs to subscribe to the face data recognized by the image based on the A specification.
[0232] S716. The management module in the intelligent perception software framework determines that an image of specification B is being subscribed, where specification B is larger than specification A.
[0233] S717. The management module in the intelligent perception software framework subscribes to the AON camera module for facial data recognized based on the B-specification image.
[0234] S718 and AON camera modules call back the B-specification image and the facial data recognized based on the B-specification image to the intelligent perception software framework.
[0235] The "Stare Without Turning Off the Screen" service needs to obtain the facial data output by the AON camera module, where the facial data can be the image recognition based on the A specification by the AON camera module. When the management module in the smart perception software framework subscribes to the facial data, it can subscribe to the specifications of the image used for identification. When only the "Stare Without Turning Off the Screen" service is running and there are no other AON services concurrently, the smart perception software framework can subscribe to the AON camera module for facial data recognized based on the A specification image. The facial data may include the judgment of whether there is a face in the image, as well as the coordinates of the face and the coordinates of the eyes when there is a face in the image.
[0236] Since the air gesture service is in the second stage of gesture recognition, the management module of the intelligent perception software framework is subscribing to images of specification B from the AON camera module at this stage, while the AON camera module can only output images of the same specification. The B specification (for example, 320*240 resolution) is larger than the A specification (for example, 160*120 resolution), so the management module in the intelligent perception software framework can subscribe to facial data according to the larger specification B. In other words, the management module in the intelligent perception software framework can subscribe to facial data recognized based on images of specification B. In this way, in addition to calling back images of specification B, the AON camera module also calls back facial data recognized based on images of specification B.
[0237] Figure 7 The timing diagram shown is only one of the scenarios of concurrent AON services. In this scenario, the "attention without turning off the screen" service is enabled in the second gesture stage. This scenario is not limited to this scenario, and other concurrent AON services can also be used. Figure 6A-6C For example, in Fig. 6A In the embodiment shown, the remote screenshot service is in operation, and the smart perception software framework can subscribe to the A-specification image of the first stage of the gesture. After the electronic device receives and responds to the user's single-click operation of the slide switch 601 to start the gaze-on screen service, the smart perception software framework can subscribe to the A-specification image and the face data recognized based on the A-specification image. Figure 6B As shown in the figure, the front camera captures the user's hand as the starting gesture, and the second stage of gesture recognition is triggered. The second stage of gesture recognition requires the use of B-specification images. The smart perception software framework can subscribe to B-specification images from the AON camera module. At the same time, the smart perception software framework also subscribes to face data based on B-specification image recognition to implement the "attention without turning off the screen" service. Figure 6C As shown, the smart perception software framework can recognize the user's gripping gesture based on the B-specification image, and then intercept the displayed smart perception setting interface 330. Finally, the smart perception software framework can continue to subscribe to the A-specification image and the face data recognized based on the A-specification image.
[0238] When the image specifications required by multiple AON services are different, the management module in the intelligent perception software framework can subscribe images or face data to the AON camera module according to the largest specification. The intelligent perception software framework can downsample large-size images to meet the needs of different AON services.
[0239] Some applications need to use the front camera to capture images, such as camera applications and instant messaging applications. If an application is using the front camera and the smart sensing application starts the AON service, the AON service will be temporarily blocked to ensure that other applications can use the front camera normally.
[0240] The following first introduces the scenario in which the AON service is enabled when other applications use the front camera.
[0241] Figure 8A-8K The user interaction diagram exemplarily shows that the AON service is enabled when other applications use the front camera.
[0242] like Fig. 8A As shown, the electronic device 100 displays the smart sensing setting interface 330, the current smart sensing application has not been started, and the front camera 193A is in a closed state. Fig. 8A The description of the smart sensing setting interface 330 can refer to the aforementioned Figure 3D The introduction of the smart perception setting interface 330 shown will not be repeated here.
[0243] like Figure 8B As shown, after the instant messaging application in the electronic device 100 receives the video call invitation, the electronic device 100 may pop up and display a video call invitation interface 801. The video call invitation interface 801 is used to accept or reject the contact's video chat request. The video call invitation interface may include an answer button 802, which is used to accept the video chat request.
[0244] The electronic device 100 may receive an operation of the user clicking the answer button 802, and in response to the operation, the electronic device 100 may start the front camera 193A and display the following information: Figure 8C The video chat interface 810 can be used to display the video screen of the contact. The video chat interface can include a preview window 811, a zoom button 812 and a hang-up button 813.
[0245] The preview window 811 may be used to display a preview of the image captured by the front camera 193A.
[0246] The zoom button 812 is used to hide the video chat interface and return to the user interface before receiving the video call invitation (i.e., the smart perception setting interface 330).
[0247] The hang-up button 813 is used to stop the currently ongoing video call.
[0248] The electronic device 100 may receive and respond to the user's operation of clicking the zoom button 812, thereby displaying Fig.8D The smart perception setting interface 330 shown. At this time, the user interface can also include a preview window 811 suspended and displayed on the smart perception setting interface 330, and the preview window 811 can display the video screen of the contact.
[0249] like Fig. 8EAs shown, the electronic device 100 can receive an operation of the user clicking the slide switch 331 of the smart sensing application, and in response to the operation, the electronic device 100 can start the smart sensing application. Figure 8F As shown, the electronic device can also receive and respond to the user's single click of the sliding switch 332 for air screenshot to start the air screenshot service. Since the front camera is occupied by the instant messaging application, the instruction to start the air screenshot service will not be sent to the smart perception software framework, but will be blocked by the conflict management module. Figure 8F , Figure 8G As shown, since the front camera 193A is occupied by the instant messaging application, the startup of the air screenshot service is blocked, so even if the front camera 193A captures a person's gesture, the air screenshot service will not be triggered. In this scenario, the AON camera module does not obtain the image captured by the front camera 193A.
[0250] like Figure 8H As shown, the electronic device 100 may receive an operation of the user clicking on the preview window 811, and in response to the operation, the electronic device 100 may display the following Figure 8I The video chat interface 810 shown. The electronic device can receive an operation in which the user clicks the hang-up button 813. In response to this operation, the electronic device 100 can end the video chat through the instant messaging application, and the instant messaging application will stop occupying the front camera 193A. The AON camera module will occupy the front camera 193A and obtain the image captured by the front camera 193A. The conflict management module will resume the startup of the air screenshot service and instruct the service management module to distribute the air screenshot service through the distributor. The intelligent perception software framework will subscribe to the images required for the air screenshot service from the AON camera module, and receive the images called back by the AON camera module for analysis and processing to determine whether the air screenshot service is triggered. Figure 8J , Figure 8K As shown, after the front camera 193A captures the movement of the user's hand from fully open to grasping, the smart perception software framework recognizes the above gesture change and determines that the air screenshot service is triggered, and then the electronic device 100 can take a screenshot of the smart perception setting interface 330 being displayed, and display a preview image 334 of the captured image in the lower left corner of the user interface.
[0251] Fig. 9 This is a timing diagram provided in an embodiment of the present application of a conflict management module processing AON services when other applications use a front camera.
[0252] like Fig. 9 As shown, when an application other than the smart sensing application in the electronic device 100 uses the front camera, the management method of the conflict management module for the AON service start instruction may include but is not limited to the following steps:
[0253] S901: The instant messaging application receives an operation in which the user accepts a video call invitation.
[0254] S902: The instant messaging application activates the front camera through the camera manager.
[0255] refer to Figure 8B In the embodiment shown, after receiving the video call invitation, the instant messaging application can receive the user's operation of clicking the answer button 802 in the video call invitation interface 801, which is used to accept the video call invitation. In response to this operation, the instant messaging application can call the interface of the camera manager to start the front camera 193A. The instant messaging application can capture images through the front camera 193A and Figure 8C The preview window 811 shown displays a preview of the image captured by the front camera 193A.
[0256] S903: The smart perception application receives an operation from the user to enable the air gesture service.
[0257] S904. The smart perception application sends an instruction to the application framework layer to enable the air gesture service.
[0258] S905. The application framework layer sends an instruction to the smart perception interface and service module to enable the air gesture service.
[0259] S906. The intelligent perception interface and service module sends an instruction to enable the air gesture service to the conflict management module.
[0260] S907. The physical conflict management module in the conflict management module determines that the front camera is occupied, blocking the opening of the air gesture service.
[0261] After the smart perception application receives the user's operation to turn on the air gesture service, the instruction can be sent to the application framework layer. After the final conflict management module receives the instruction to turn on the air gesture service, the physical conflict management module in the conflict management module can determine that the front camera has been occupied by the instant messaging application. In order to prevent the AON camera module and the instant messaging application from occupying the front camera after the smart perception application turns on the air gesture service, the physical conflict management module can block the opening of the air gesture service, that is, the air gesture service will not be distributed through the distributor for the time being.
[0262] refer to Fig.8D , Fig. 8EIn the embodiment shown, the user can enable the air gesture service by clicking the sliding switch 332 of the air screenshot service. In response to this operation, the smart perception application can send an instruction to enable the air gesture service to the application framework layer, and the conflict management module can temporarily block the air gesture service after receiving the instruction. Figure 8F , Figure 8G As shown, since the air gesture service is blocked, even if the front camera 193A captures the image of the gesture used to trigger the air screenshot service, the air screenshot service will not be triggered.
[0263] S908: The instant messaging application receives an operation from the user to stop the video call.
[0264] S909. Stop using the front camera for instant messaging applications.
[0265] S910. The physical conflict management module in the conflict management module determines that the instant messaging application stops occupying the front camera and resumes the air gesture service.
[0266] S911. The conflict management module sends an instruction to the intelligent perception software framework to enable the air gesture service.
[0267] S912. The intelligent perception software framework subscribes to the AON camera module for images required for the air gesture service.
[0268] After receiving the user's operation to stop the video call, the instant messaging application can stop the video call and stop occupying the front camera. After the physical conflict management module in the conflict management module determines that the instant messaging application stops occupying the front camera, the conflict management module can instruct the service management module to send the instruction to start the air gesture service to the screen light service client.
[0269] refer to Figure 8I In the illustrated embodiment, the user's operation of stopping the video call may be the user clicking the hang-up button 813, in response to which the instant messaging application will stop occupying the front camera. After the conflict management module senses that the instant messaging application stops occupying the front camera, it can resume starting the air gesture service. After receiving the instruction to start the air gesture service, the intelligent perception software framework can subscribe to the AON camera module for the images required for the air gesture service, and at this time the AON camera module will occupy the front camera. Figure 8J , Figure 8K As shown, after the air gesture service is resumed and started, the AON camera module can obtain the image of the gesture captured by the front camera 193A for triggering the air screenshot service. The intelligent perception software framework can determine that the air screenshot service is triggered based on the image, and then the electronic device 100 can perform a screenshot operation.
[0270] In some embodiments, when the AON camera module occupies the front camera 193A, if other applications need to use the front camera 193A, the AON camera module will stop occupying the front camera 193A. After other applications stop occupying the front camera 193A, the AON camera module receives the image of the front camera 193A again, processes the image, and calls back to the smart perception software framework. In other words, the priority of the smart perception application to use the front camera 193A is lower than the priority of other applications (such as camera applications, instant messaging applications, etc.) to use the front camera 193A. This can prevent the AON service from occupying the front camera, causing other applications to fail to operate normally, affecting the user experience.
[0271] It should be noted that Fig. 9 Taking the example of when the instant messaging application occupies the front camera and the smart perception application starts the air gesture service to introduce the method of the conflict management module to handle physical conflicts. It is not limited to instant messaging applications, but can also be other applications that use the front camera. At the same time, it is not limited to the air gesture service, but can also be other AON services. When other applications occupy the front camera, the method of starting the AON service can refer to the method of starting the air gesture service when the above-mentioned instant messaging application occupies the front camera, and will not be repeated here.
[0272] The following describes the business management method provided by the embodiment of the present application.
[0273] Fig.10 is a flow chart of the service management method provided in the embodiment of the present application, such as Fig.10 As shown, the method may include but is not limited to:
[0274] S1001. The electronic device turns on a first function. When the first function is turned on, the first camera is in an on state. The first function is used to obtain image data collected by the first camera. The operation of the first application and the second application is based on the image data.
[0275] In the embodiment of the present application, the front camera may be referred to as the first camera, and the first function may be the AON function. Turning on the AON function actually means turning on the function of the AON camera module to obtain the image data collected by the front camera. When the AON function is turned on, the AON camera module will instruct the front camera to remain turned on. After the ISP processes the image data collected by the front camera, the AON camera module can obtain the image data processed by the ISP.
[0276] The electronic device may receive and start the first function in response to the user operation. Figure 3DIn the embodiment shown, after the electronic device receives the operation of the user clicking the smart sensing sliding switch 331, it can start the first function in response to the operation. It is not limited to the above operation, and can also be other operations. For example, when the AON service is not turned on, the electronic device 100 can also start the first function after receiving the operation of the user turning on the first AON service. Figure 3E In the illustrated embodiment, when the AON screen-on service and the AON screen-off service are not turned on, after the electronic device 100 receives an operation of clicking the sliding switch 332 for the air screenshot service, it can start the first function in response to the operation and then turn on the air screenshot service.
[0277] S1002. The electronic device receives an instruction to start a first application at a first time point, and the electronic device receives an instruction to close a second application at a second time point, the first time point is earlier than the second time point, the first application uses image data when the display screen is in a first on-off state, and the second application uses image data when the display screen is in a second on-off state.
[0278] The first on-off state includes: a screen-on state and a screen-off state, the second on-off state also includes a screen-on state and a screen-off state, and the first on-off state is different from the second on-off state. For example, when the first on-off state is a screen-on state, the second on-off state is a screen-off state.
[0279] In the embodiment of the present application, the low power consumption subsystem may also be referred to as the first subsystem layer. The AON camera module in the low power consumption subsystem layer may be referred to as the first module. The operation of the first application and the second application is based on the image data after the AON camera module processes the image data collected by the front camera.
[0280] The first application may include one or more applications, each of which is used to implement a single AON service when the display screen is in a first on-off state. The second application may include one or more applications, each of which is used to implement a single AON service when the display screen is in a second on-off state. Taking the first on-off state as the on-screen state as an example, the first application may include a smart code scanning application, a smart horizontal and vertical screen application, an auxiliary photo application, etc., wherein the smart code scanning application may be an application for implementing a smart code scanning service (function), and the smart horizontal and vertical screen application may be an application for implementing a smart horizontal and vertical screen service (function). The above-mentioned smart code scanning application (smart code scanning service), smart horizontal and vertical screen application (smart horizontal and vertical screen service), and auxiliary photo application (auxiliary photo service) can be regarded as sub-applications in the smart perception application, and the sub-application is used to implement a single service function.
[0281] Specifically, the electronic device 100 receives the instruction to start the first application at the first time point and receives the instruction to close the second application at the second time point. The above instructions can be issued by the electronic device when the display screen switches between the on screen state and the off screen state. The electronic device can receive and respond to the user's operation of clicking the power button 190A to implement the switch between the on screen state and the off screen state. For example, from FIG. 3I to FIG. 3J In the embodiment shown, the electronic device receives and responds to the user's operation of clicking the power button 190A, and then the display screen switches from the on screen state to the off screen state. The electronic device will issue an instruction to turn on the AON off screen service and turn off the AON on screen service. Figure 3L to Figure 3M In the embodiment shown, when the display screen of the electronic device switches from the off screen state to the on screen state, the electronic device may issue an instruction to turn on the AON on screen service and turn off the AON off screen service. Figure 3F In the smart perception setting interface 330 shown, the electronic device receives the AON service corresponding to the AON service option selected by the user.
[0282] S1003: The electronic device starts a first application, and when the first application is started, the second application is set to be closed.
[0283] When the first on-off state is the on-screen state and the second on-off state is the off-screen state, if the conflict management module in the electronic device determines that the second application is in the on state at the first time point, the first application is blocked from being opened; when the conflict management module in the electronic device closes the second application and then unblocks the first application, the second application is closed later than the second time point. Figure 3L-3M In the embodiment shown, the first application may be an air screenshot application, which is used to implement the air screenshot service function, and the second application may be a face sensing screen off display application, which is used to implement the face sensing screen off display service function. When the electronic device switches from the screen-off state to the screen-on state, if the conflict management module first receives an instruction to turn on the air screenshot service, the conflict management module may first block the turning on of the air screenshot service instruction. After the conflict management module receives an instruction to turn off the face sensing screen off display service at a second time point, it may instruct the screen off service client to turn off the face sensing screen off display service, and the AON camera module may stop performing face recognition on the image (reference Figure 5 After the face screen display service is closed, the conflict management module can restore the blocked startup instruction of the air screenshot service (refer to Figure 516-18 in the timing diagram shown in FIG. 1 ). When the air screenshot application is turned on, the AON camera module of the smart perception software framework subscribes to the image data required by the air screenshot service. When the air screenshot service is triggered, the message will be called back to the air screenshot application.
[0284] When the first on-off state is the screen-off state and the second on-off state is the screen-on state, if the conflict management module in the electronic device determines that the second application is in the open state at the first time point, the conflict management module first closes the second application. When the second application is closed, the second application stops subscribing to the image data of the AON camera module (refer to Figure 4 Steps S413 and S414 in the timing diagram shown in FIG. Figure 3I-3J Taking the shown embodiment as an example, the first application may be a face sensing and screen-off display application, and the second application may be an air screenshot application. When the electronic device switches from a screen-on state to a screen-off state, if the conflict management module first receives an instruction to enable the face sensing and screen-off display service, the conflict management module may directly send an instruction to close the air screenshot application to the smart perception software framework. The above instruction to close the air screenshot application is also an instruction to close the air screenshot service. The smart perception software framework may then cancel the subscription to the AON camera module for the image data required for the air screenshot service. When the conflict management module receives an instruction to close the second application (i.e., an instruction to close the air screenshot application) at a second time point, it will no longer process the instruction (reference Figure 4 (steps S417-S420 in the timing diagram shown).
[0285] In some embodiments, the first application subscribes to a plurality of different specifications of image data to the first function, and the operation of the first application is based on the image data of the first specification, and the image data of the first specification is the image data of the maximum specification selected from the image data of the plurality of different specifications. Take the AON screen light service provided by the first application as an example. For example, the second stage of the air gesture service needs to subscribe to the image data of 320*240 resolution from the AON camera module, and the recognition of the smart code scanning needs to subscribe to the image data of 160*120 resolution from the AON camera module. The smart perception software framework can select the image data of the maximum specification (i.e., 320*240 resolution) from the above two resolutions and subscribe to the AON camera module. The above-mentioned 320*240 resolution can be called the first specification. When the second application provides a screen-off service, if multiple screen-off services need to subscribe to image data of different specifications from the AON camera module, the AON camera module can process multiple screen-off services according to the image data of the maximum specification.
[0286] In some embodiments, the operation of the first application or the second application is based on the face recognition result, the first application subscribes to the face recognition result to the first function, and the first function is also used to output the face recognition result based on the face image data. That is to say, the AON camera module can also recognize the face recognition result based on the image data, and the face recognition result here is not limited to: whether there is a face in the image, and when there is a face in the image, the coordinates of the face, the coordinates of the eyes, etc. Figure 6A-6C In the embodiment shown, the first application may include a non-extinguishing screen application and a screen capture application. The non-extinguishing screen application and the screen capture application need to subscribe to image data of different specifications from the AON camera module. For example, the non-extinguishing screen application needs to subscribe to the face data based on image recognition with a resolution of 160*120 from the AON camera module, and the screen capture application needs to subscribe to image data with a resolution of 320*240 from the AON camera module in the second stage of gesture recognition. The smart perception software framework in the electronic device can subscribe to the AON camera module according to the image data of the maximum specification (i.e., image data with a resolution of 320*240). Here, the resolution of 320*240 is the first specification. In the embodiment of the present application, the smart perception software framework may also be referred to as the second module.
[0287] In some embodiments, the electronic device also includes a third application, and the operation of the third application is also based on the image data collected by the first camera. The priority of the third application is higher than the priority of the first application and the second application. Before the electronic device activates the first function, the method also includes: the electronic device determines that the first camera is not occupied by the third application. Among them, the third application can be an application based on the first camera. The priority here refers to the priority of using the front camera. For example, the third application can be a software module that implements a video chat service in an instant messaging application. The electronic device will only activate the first function if it determines that the first camera is not occupied by the third application. Reference Figure 8A-8G In the embodiment shown, when the instant messaging application uses the front camera, if the electronic device 100 receives a user's instruction to start the air screenshot service, since the implementation of the air screenshot service is based on the image data of the front camera obtained by the AON camera module, the conflict module will first block the start of the air screenshot service, so that the AON camera module will not compete with the instant messaging application for the front camera. When the electronic device determines that the front camera is not occupied by the instant messaging application, it will resume the subscription of the AON camera module to the image data of the front camera (refer to Figure 8H-8K embodiment shown).
[0288] In some embodiments, after the electronic device starts the first function, the method further includes: when the electronic device detects that the third application is started, the electronic device stops providing the first function with the data collected by the first camera. In other words, when the electronic device has started the AON service, when other applications use the front camera, the electronic device can temporarily stop the AON service, and the AON camera module no longer obtains the data collected by the first camera.
[0289] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0290] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.
[0291] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integration. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.
[0292] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.
Claims
1. A service management method, the method being applied to an electronic device, It is characterized in that The electronic device includes a first application and a second application, and the electronic device also includes a display screen and a first camera. The method includes: The electronic device turns on a first function, and when the first function is turned on, a first camera of the electronic device is in an on state, the first function is used to obtain image data collected by the first camera, and the operation of the first application and the second application is based on the image data; The electronic device receives an instruction to start the first application at a first time point, and receives an instruction to close the second application at a second time point, the first time point is earlier than the second time point, the first application uses the image data when the display screen is in a first on-off state, and the second application uses the image data when the display screen is in a second on-off state; When the first on-off state is the screen-off state and the second on-off state is the screen-on state, the electronic device determines that the second application is in the on state at the first time point, closes the second application, and the time point of closing the second application is earlier than the second time point; In the case where the first on-off state is a screen-on state and the second on-off state is a screen-off state, the electronic device determines that the second application is in an on state at the first time point, blocks the first application, and unblocks the first application after the electronic device closes the second application, and the time point when the second application is closed is later than the second time point; The electronic device starts the first application, and when the first application is started, the second application is set to a closed state.
2. The method according to claim 1, It is characterized in that The electronic device further includes a power button. Before the electronic device receives an instruction to start the first application at the first time point, the method further includes: the electronic device receives an operation of a user clicking the power button.
3. The method according to claim 1, It is characterized in that The method also includes: the first application subscribes to image data of multiple different specifications to the first function, the operation of the first application is based on image data of a first specification, and the image data of the first specification is image data of the largest specification selected from image data of multiple different specifications.
4. The method according to claim 1, It is characterized in that The method also includes: the second application subscribes to image data of multiple different specifications to the first function, the second application runs based on image data of a second specification, and the image data of the second specification is image data of the largest specification selected from the image data of multiple different specifications.
5. The method according to any one of claims 1 to 4, It is characterized in that The operation of the first application or the second application is based on a face recognition result, the first application subscribes to the face recognition result to the first function, and the first function is also used to output the face recognition result based on the image data.
6. The method according to any one of claims 1 to 4, It is characterized in that The electronic device also includes a third application, and the operation of the third application is also based on the image data collected by the first camera. The priority of the third application is higher than the priority of the first application and the second application. Before the electronic device turns on the first function, the method also includes: the electronic device determines that the first camera is not occupied by the third application.
7. The method according to claim 6, It is characterized in that After the electronic device starts the first function, the method further includes: the electronic device detects that the third application is started, and the electronic device stops providing the data collected by the first camera to the first function.
8. The method according to claim 1, It is characterized in that The electronic device includes an application layer, a hardware abstraction layer, a first subsystem layer and a hardware layer, the application layer includes the first application and the second application, the hardware layer includes the display screen and the first camera, the hardware abstraction layer includes a conflict management module, the first subsystem layer includes a first module, when the first function is turned on, image data collected by the first camera is sent to the first module, and the operation of the first application and the second application is based on the image data processed by the first module.
9. The method according to claim 8, It is characterized in that The electronic device receives an instruction to start the first application at a first time point, and the electronic device receives an instruction to close the second application at a second time point, specifically including: The conflict management module in the hardware abstraction layer receives an instruction to start the first application at the first time point, and the conflict management module receives an instruction to close the second application at the second time point.
10. The method according to claim 9, It is characterized in that The first on-off state is a screen-on state, and the second on-off state is a screen-off state. When the electronic device starts the first application, the method further includes: The conflict management module determines that the second application is in an open state at the first time point, and blocks the first application; The conflict management module unblocks the first application after closing the second application, and the time point at which the second application is closed is later than the second time point.
11. The method according to claim 9, It is characterized in that The first on-off state is a screen-off state, and the second on-off state is a screen-on state. When the electronic device starts the first application, the method further includes: The conflict management module determines that the second application is in an open state at the first time point, closes the second application, and the time point of closing the second application is earlier than the second time point.
12. The method according to any one of claims 8 to 11, It is characterized in that The first subsystem layer also includes a second module, and the method also includes: the first application subscribes to image data of multiple different specifications to the first module, the operation of the first application is based on image data of first specifications, and the image data of the first specification is image data of the largest specification selected by the second module from image data of multiple different specifications.
13. The method according to claim 12, It is characterized in that The method also includes: the second application subscribes to image data of multiple different specifications to the first function, the operation of the second application is based on image data of second specifications, and the image data of the second specification is image data of the largest specification selected by the second module from image data of multiple different specifications.
14. The method according to any one of claims 8 to 11, It is characterized in that The operation of the first application or the second application is based on a face recognition result, the first application subscribes to the face recognition result from the first module, and the first function also outputs the face recognition result based on the image data through the first module.
15. The method according to any one of claims 8 to 11, It is characterized in that The application layer also includes a third application, and the operation of the third application is also based on the image data collected by the first camera. The priority of the third application is higher than the priority of the first application and the second application. Before the electronic device turns on the first function, the method also includes: the conflict management module determines that the first camera is not occupied by the third application.
16. The method according to claim 15, It is characterized in that After the electronic device starts the first function, the method further includes: when the electronic device detects that the third application is started, the first module stops acquiring data collected by the first camera.
17. The method according to any one of claims 1, 8, 9 and 10, It is characterized in that The services of the first application include: smart code scanning service, air gesture service, smart horizontal and vertical screen service, assisted photo taking service, staring ringtone reduction service and staring without turning off the screen service; the services of the second application include: face sensing and screen turning off service.
18. The method according to any one of claims 1, 8, 9 and 10, It is characterized in that The services of the first application include: face sensing and screen-off display service; the services of the second application include: smart code scanning service, air gesture service, smart horizontal and vertical screen service, assisted photo taking service, staring ringtone reduction service and staring without turning off the screen service.
19. The method according to any one of claims 1 to 4, 8 to 11, It is characterized in that The first camera is a front camera.
20. An electronic device, It is characterized in that The electronic device includes: a display screen, a first camera, a memory, and a processor coupled to the memory; the display screen is used to display an interface, the first camera is used to collect image data, the memory stores a computer program, and when the processor executes the computer program, the electronic device implements the method as described in any one of claims 1 to 19.
21. A computer-readable storage medium comprising computer instructions, It is characterized in that When the computer instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 19.
