Data processing method and apparatus, device, and storage medium

Abstract

The application provides a data processing method and device, equipment and a storage medium. The method comprises: performing first identification on to-be-identified data to obtain a first identification result; if the first identification result does not satisfy a trigger condition of second identification, performing the first identification on next to-be-identified data; and if the first identification result satisfies the trigger condition, triggering the second identification. The second identification is used to obtain a second identification result based on at least one to-be-identified data, and the identification accuracy of the second identification is higher than that of the first identification.

Description

Technical Field

[0001] This application relates to electronic technology, including but not limited to data processing methods, apparatus, devices, and storage media. Background Technology

[0002] like Figure 1 As shown, contextual intelligence, driven by business needs, transforms data collected by data acquisition devices (such as image sensors, microphones, accelerometers, gyroscopes, and temperature sensors) into contextual information through data acquisition devices and algorithms. This results in a deep understanding of the user and proactive service delivery. Contextual awareness has also become a trend in current smart devices (such as smartphones, tablets, and in-vehicle devices).

[0003] Because context-aware services require data collectors to be always on (AON), achieving low power consumption on smart devices is a major challenge, especially for image-based services, which consume more power than other data collectors (such as accelerometers, gyroscopes, and microphones). However, some solutions sacrifice data recognition accuracy while reducing power consumption. Summary of the Invention

[0004] In view of this, the data processing method, apparatus, device, and storage medium provided in this application can reduce power consumption without sacrificing the recognition accuracy of the data to be identified.

[0005] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: performing first-level identification on data to be identified to obtain a first identification result; if the first identification result does not meet the triggering condition for second-level identification, performing the first-level identification on the next data to be identified; if the first identification result meets the triggering condition, triggering the second-level identification, wherein the second-level identification is used to obtain a second identification result based on at least one of the data to be identified, and the identification accuracy of the second-level identification is higher than the identification accuracy of the first-level identification.

[0006] The data processing apparatus provided in this application includes: an identification module, configured to perform first-level identification on the data to be identified to obtain a first identification result; and configured to perform the first-level identification on the next data to be identified if the first identification result does not meet the triggering condition for second-level identification; and a triggering module, configured to trigger the second-level identification if the first identification result meets the triggering condition, wherein the second-level identification is used to obtain a second identification result based on at least one of the data to be identified, and the accuracy of the second-level identification is higher than the accuracy of the first-level identification.

[0007] The electronic device provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the method described in this application.

[0008] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the method described in this application embodiment.

[0009] In this embodiment, a data processing method is provided. First, the data to be identified is subjected to primary identification to obtain a first identification result. If the first identification result does not meet the triggering condition for secondary identification, secondary identification is not performed, and the method returns to perform primary identification on the next data to be identified. If the first identification result meets the triggering condition for secondary identification, secondary identification is triggered. The function of secondary identification is the same as that of primary identification, both of which are used to obtain identification results based on the data to be identified. For example, secondary identification is used to obtain a second probability of belonging to a specific category based on at least one data to be identified. However, the identification accuracy of secondary identification is higher than that of primary identification. In this way, while reducing power consumption, or while meeting low power consumption requirements, secondary identification can use a higher accuracy model without sacrificing identification accuracy.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0012] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0013] Figure 1 This is a schematic diagram of a context-aware application scenario;

[0014] Figure 2 A schematic diagram illustrating the implementation flow of the data processing method provided in the embodiments of this application;

[0015] Figure 3A schematic diagram illustrating the implementation flow of another data processing method provided in an embodiment of this application;

[0016] Figure 4 A schematic diagram illustrating the implementation flow of another data processing method provided in this application embodiment;

[0017] Figure 5 This is a schematic diagram of the structure of a mobile terminal provided in an embodiment of this application;

[0018] Figure 6 A schematic diagram illustrating the relationship between confidence level and frame rate provided in this application embodiment;

[0019] Figure 7 This is a schematic diagram of the structure of the data processing device according to an embodiment of this application;

[0020] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0024] It should be noted that the terms "first, second, third" used in the embodiments of this application do not represent a specific order of objects. It is understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0025] This application provides a data processing method applied to an electronic device. This electronic device can be of various types with data processing capabilities, such as mobile phones, tablets, in-vehicle devices, laptops, personal computers, and servers. The functions implemented by this method can be achieved by a processor in the electronic device calling program code. The program code can be stored in a computer storage medium. Therefore, the electronic device includes at least a processor and a storage medium.

[0026] Figure 2 This is a schematic diagram illustrating the implementation flow of the data processing method provided in the embodiments of this application, as shown below. Figure 2 As shown, the method may include the following steps 201 to 203:

[0027] Step 201: Perform first-level recognition on the data to be recognized to obtain the first recognition result.

[0028] The data to be identified can be data collected by a data acquisition device inside an electronic device, or data output after preprocessing of the data collected by the data acquisition device; the data to be identified can also be data received by the electronic device from other devices. In this application, there is no limitation on the type of data to be identified; it can be of various types. In other words, the type of data to be identified may differ for different application scenarios. For example, in a scenario of monitoring whether a smartphone is being spied on, the data to be identified is image data; in a service experience scenario based on mobile phone location prediction, the data to be identified is the location information of the mobile phone; in a voice recognition service scenario, the data to be identified is voice data; and in a service scenario based on gait recognition and travel status detection, the data to be identified is data such as acceleration and angular velocity.

[0029] In this application, the method used for primary recognition is not limited and can be varied. For example, a pre-trained neural network model based on a specific scenario can be used for primary recognition of the data to be recognized. Alternatively, other algorithms can be used to achieve primary recognition.

[0030] Step 202: Determine whether the first recognition result meets the triggering condition for secondary recognition; if yes, proceed to step 203; otherwise, return to step 201.

[0031] Understandably, if the first recognition result does not meet the triggering conditions for the second-level recognition, the first-level recognition is performed on the next data to be recognized, instead of continuing further processing. That is, the relevant procedures for the second-level recognition are no longer executed, thereby saving power consumption while meeting the service quality requirements of the scenario.

[0032] In this application, there is no limitation on whether the first recognition result includes one or more recognition sub-results. It can be one, that is, a sub-result obtained by performing first-level recognition based on one piece of data to be recognized (e.g., an image); or it can be two or more. For example, the first recognition result includes: a sub-result obtained by performing first-level recognition based on the data to be recognized collected at a first time, and a sub-result obtained by performing first-level recognition based on the data to be recognized collected at a second time. Of course, for the above different situations, the corresponding triggering conditions for second-level recognition can also be different.

[0033] In this application, the representation method of the first identification result is not limited and can be various. However, different representation methods may correspond to different triggering conditions. For cases where the first identification result includes an identification sub-result, for example, in some embodiments, the first identification result represents the final judgment result. For instance, in a scenario monitoring whether a smartphone is being spied on, the first identification result represents whether the phone is being spied on, and the corresponding triggering condition for secondary identification is that the phone is being spied on. In other embodiments, the first identification result represents a first probability of belonging to a specific category. For instance, in a scenario monitoring whether a smartphone is being spied on, the first identification result represents a first probability of the phone being spied on, and the corresponding triggering condition for secondary identification is that the first probability is greater than a first threshold; where, in this scenario, the specific category is that the phone is being spied on.

[0034] When the first identification result includes two or more identification sub-results, the triggering conditions for the corresponding secondary identification also differ depending on the representation method of the identification sub-results. For example, in some embodiments, the identification sub-results in the first identification result represent the final judgment result. For instance, in a scenario of monitoring whether a smartphone is being spied on, the identification sub-result represents whether the phone is being spied on, and the corresponding triggering condition for secondary identification is that the number of identification sub-results indicating that the phone is being spied on is greater than or equal to a correspondingly set threshold. In other embodiments, the identification sub-result represents the first probability of belonging to a specific category. For instance, in a scenario of monitoring whether a smartphone is being spied on, the identification sub-result represents the first probability of the phone being spied on, and the corresponding triggering condition for secondary identification is that the number of identification sub-results with a first probability greater than a first threshold is greater than or equal to a correspondingly set threshold; where, in this scenario, the specific category is that the phone is being spied on.

[0035] Step 203: Trigger the secondary recognition, which is used to obtain a second recognition result based on at least one of the data to be recognized. The recognition accuracy of the secondary recognition is higher than that of the primary recognition.

[0036] It should be noted that, in this application, the processors executing the first-level identification program and the second-level identification program are not limited to being the same processor or different processors.

[0037] In some embodiments, the processor executing the first-level recognition program is a second processor, and the processor executing the second-level recognition program is a first processor. The power consumption of the first processor can be the same as or different from that of the second processor. In some implementations, the power consumption of the first processor is higher than that of the second processor. For example, the second processor is a front-end image processing chip, and the first processor is the main chip. The second processor executes steps 201 and 202, storing the first recognition result obtained from the first-level recognition and the corresponding data to be recognized, so that the first processor can subsequently perform second-level recognition based on at least one of the stored data to be recognized.

[0038] In some embodiments, the first identification result includes a first probability of belonging to a specific category, and the second processor may store both the first identification result obtained from the first identification and the corresponding data to be identified; the second processor may also store only the first identification result with a first probability greater than a second threshold and the corresponding data to be identified; wherein, the second threshold is less than or equal to the first threshold.

[0039] In some embodiments, the second processor may send to the first processor the stored data to be identified that has a first probability greater than or equal to a second threshold, so that the first processor can perform the secondary identification based on the data to be identified that has a first probability greater than or equal to the second threshold.

[0040] Understandably, the second threshold is less than or equal to the first threshold in the triggering conditions of the second-level recognition. Thus, if the value of the second threshold is equal to the first threshold, the first processor will process the data to be recognized with a first probability greater than the first threshold when performing the second-level recognition, thereby reducing the amount of data processed for the second-level recognition. If the value of the second threshold is less than the first threshold, the first processor will process the data to be recognized with a first probability less than or equal to the first threshold when performing the second-level recognition, thereby compensating for the situation of misjudgment in the first-level recognition.

[0041] In this embodiment, a data processing method is provided. First, the data to be identified is subjected to primary identification to obtain a first identification result. If the first identification result does not meet the triggering condition for secondary identification, secondary identification is not performed, and the method returns to perform primary identification on the next data to be identified. If the first identification result meets the triggering condition for secondary identification, secondary identification is triggered. The function of secondary identification is the same as that of primary identification, both of which are used to obtain identification results based on the data to be identified. For example, secondary identification is used to obtain a second probability of belonging to a specific category based on at least one data to be identified. However, the identification accuracy of secondary identification is higher than that of primary identification. In this way, while reducing power consumption, or while meeting low power consumption requirements, secondary identification can use a higher accuracy model without sacrificing identification accuracy.

[0042] For example, in some embodiments, the neural network model used for secondary recognition is more complex in structure than that used for primary recognition. For instance, the neural network model used for primary recognition may be a lightweight model. Of course, primary recognition may also not use a neural network model, but rather a simpler recognition algorithm, while secondary recognition uses a high-precision neural network model.

[0043] This application embodiment further provides a data processing method. Figure 3 This is a schematic diagram illustrating the implementation flow of the data processing method provided in the embodiments of this application, as shown below. Figure 3 As shown, the method may include the following steps 301 to 303:

[0044] Step 301: Perform first-level identification on the data to be identified to obtain the first probability of belonging to a specific category.

[0045] In some embodiments, the electronic device can cache the first-level recognition result and the corresponding data to be recognized for use in subsequent second-level recognition. The electronic device can cache each recognition result, or it can cache a portion of the recognition results, for example, caching the data to be recognized with a first probability greater than or equal to a second threshold.

[0046] Step 302: Determine whether the first probability is less than or equal to the first threshold; if yes, return to step 301; otherwise, proceed to step 303.

[0047] Step 303: Trigger secondary recognition, which is used to obtain a second probability of belonging to the specific category based on at least one of the stored data to be identified, and the accuracy of the secondary recognition is higher than that of the primary recognition.

[0048] In some embodiments, the secondary identification is used to obtain a second identification result based on the stored data to be identified where the first probability is greater than or equal to a second threshold, the second identification result including a second probability of belonging to the specific category; wherein the second threshold is less than or equal to the first threshold.

[0049] Understandably, the second threshold is less than or equal to the first threshold in the triggering conditions of the second-level recognition. Thus, if the value of the second threshold is equal to the first threshold, then during the second-level recognition, the data to be recognized with a first probability greater than the first threshold is processed, which can reduce the amount of data processed for the second-level recognition. If the value of the second threshold is less than the first threshold, then during the second-level recognition, the data processed includes the data to be recognized with a first probability less than or equal to the first threshold. In this way, the situation of misjudgment in the first-level recognition can be compensated.

[0050] This application further provides a data processing method. Figure 4 This is a schematic diagram illustrating the implementation flow of the data processing method provided in the embodiments of this application, as shown below. Figure 4 As shown, the method may include the following steps 401 to 405:

[0051] Step 401: The second processor performs first-level identification on the data to be identified to obtain the first probability of belonging to a specific category.

[0052] In some embodiments, the data to be identified can be the data output by the data collector. The data collector can first detect the data to be identified, and if the detection result meets the conditions, it will trigger the second processor to execute step 401; in this way, power consumption can be saved.

[0053] For example, in a scenario where a smartphone is being spied on, the data collector can first use its motion detection (MD) function to detect the presence of moving objects in the environment. If a moving object is found, the collected data to be identified is transmitted to a second processor, triggering the second processor to perform primary identification on the data. If no moving object is found, the system continues to detect the presence of moving objects in the environment according to the monitoring rules. In this application scenario, the data to be identified can be high-resolution image data (e.g., RGB images) or low-resolution image data (e.g., grayscale images).

[0054] Step 402: The second processor caches the data to be identified and the corresponding first probability.

[0055] Of course, in other embodiments, the second processor may also cache only the data to be identified that has a first probability greater than a second threshold, so that the first processor can use it when performing secondary identification.

[0056] Step 403: The second processor determines whether the first probability is less than or equal to the first threshold; if so, proceed to step 404; otherwise, return to step 401.

[0057] Step 404: The second processor sends the data to be identified from the stored data to be identified, in which the first probability is greater than or equal to the second threshold, to the first processor; wherein the second threshold is less than or equal to the first threshold.

[0058] In some embodiments, the power consumption of the first processor is greater than that of the second processor. For example, in image data-based applications, the second processor is a lightweight front-end image processing chip, while the first processor is the main processor, and its power consumption is higher than that of the lightweight front-end image processing chip.

[0059] Step 405: The first processor performs the secondary identification based on the data to be identified whose first probability is greater than or equal to the second threshold, to obtain a second probability belonging to the specific category; wherein, the identification accuracy of the secondary identification is higher than that of the primary identification.

[0060] In this embodiment, the second processor first performs primary recognition on the data to be identified. If the result of the primary recognition meets the triggering condition for secondary recognition (in this case, the probability is greater than a first threshold), then secondary recognition is performed. Otherwise, secondary recognition is not performed, and the processor returns to perform primary recognition on the next data to be identified. This saves a certain amount of power consumption, thus creating conditions for using a higher-precision recognition algorithm for secondary recognition. In other words, power consumption is saved without sacrificing the recognition accuracy of secondary recognition.

[0061] In some embodiments, the neural network model used for secondary recognition has a higher recognition accuracy than the neural network used for primary recognition. Thus, the former often has a higher model complexity than the latter. However, the data processing method provided in this application saves power consumption, thereby creating conditions for using a higher-precision neural network model for secondary recognition.

[0062] To further conserve the overall power consumption of electronic devices, in some embodiments, the method further includes: if the number of times the first probability is less than or equal to the first threshold within a first time period is greater than or equal to a third threshold, or if the first probability is less than or equal to the first threshold for N1 consecutive times, obtaining a third probability based on at least one first probability less than or equal to the first threshold; adjusting the current acquisition frequency of the data to be identified based on the third probability; wherein the third probability is positively correlated with the current acquisition frequency, and N1 is greater than 0.

[0063] Here, the first probability refers to the probability of belonging to a specific category, which in turn refers to the target category detected in the corresponding application scenario. For example, in a scenario where a smartphone is being spied on, the specific category is being spied on, and the first probability is the probability that a spying event has occurred.

[0064] The so-called positive correlation between the third probability and the current sampling frequency means that the higher the third probability, the higher the corresponding sampling frequency; and the lower the third probability, the lower the corresponding sampling frequency. Of course, this is an adjustment strategy in the absence of a specific category, such as when no spying event is detected.

[0065] In other words, if the number of times the first probability is less than or equal to the first threshold within the first time period is greater than or equal to the third threshold, or if the first probability is less than or equal to the first threshold for N1 consecutive times, it indicates that a specific category does not exist in the current environment, such as the absence of peeping events. Therefore, the collection frequency of the data to be identified can be adjusted based on the magnitude of the third probability. The smaller the third probability, the lower the probability of belonging to a specific category, thus the current collection frequency can be lowered to save power. If the third probability is larger, it indicates that a specific category may appear, such as the possibility of peeping events, in which case the current collection frequency can be increased to detect the specific category in a timely manner, thereby improving response speed.

[0066] In this application, N1 can be any value, without restriction, and can be set according to the indicator requirements of the service scenario. The third threshold is also arbitrary and can be set according to the indicator requirements of the service scenario.

[0067] In some embodiments, the method further includes: if the number of times the first probability is greater than the first threshold within a second time period is greater than or equal to a fourth threshold, or if the first probability is greater than the first threshold for N2 consecutive times, obtaining a fourth probability based on at least one first probability greater than the first threshold; adjusting the current acquisition frequency of the data to be identified based on the fourth probability; wherein the fourth probability is negatively correlated with the current acquisition frequency, and N2 is greater than 0.

[0068] Similarly, here, the first probability refers to the probability of belonging to a specific category, which in turn refers to the target category detected in the corresponding application scenario. For example, in a scenario where a smartphone is being spied on, the specific category is being spied on, and the first probability is the probability that a spying event has occurred.

[0069] In contrast to the previous embodiment, the so-called fourth probability is negatively correlated with the current sampling frequency, meaning that the higher the fourth probability, the higher the corresponding sampling frequency; and the lower the fourth probability, the lower the corresponding sampling frequency. Of course, this is an adjustment strategy in the case of a specific category, such as when a spying event is detected.

[0070] In other words, if the number of times the first probability is greater than the first threshold within the second time period is greater than or equal to the fourth threshold, or if the first probability is greater than the first threshold N2 consecutive times, it indicates that a specific category exists in the current environment, such as a spying event. Therefore, the collection frequency of the data to be identified can be adjusted according to the magnitude of the fourth probability. The higher the fourth probability, the greater the probability of belonging to a specific category, thus the current collection frequency can be lowered to save power. This is because when a specific category has been detected with a high probability, lowering the collection frequency will not affect service quality but will save power. Conversely, a lower fourth probability requires further improvement in recognition accuracy, thus the current collection frequency should be increased.

[0071] In this application, N2 can be any value, without restriction, and can be set according to the indicator requirements of the service scenario. The same applies to the fourth threshold; it can also be any value and can be set according to the indicator requirements of the service scenario.

[0072] In some embodiments, the method further includes: if the number of times the first probability is less than or equal to the first threshold within a third time period is greater than or equal to a fifth threshold, or if the first probability is less than or equal to the first threshold for N3 consecutive times, collecting data to be identified at a first resolution; if the number of times the first probability is greater than the first threshold within a fourth time period is greater than or equal to a sixth threshold, or if the first probability is greater than the first threshold for N4 consecutive times, collecting data to be identified at a second resolution; wherein N3 and N4 are both greater than 0, and the first resolution is less than the second resolution.

[0073] Similarly, here, the first probability refers to the probability of belonging to a specific category, and the specific category refers to the target category detected in the corresponding application scenario. Understandably, if a specific category does not exist in the detection environment—for example, if the number of times the first probability is less than or equal to the first threshold within the third time period is greater than or equal to the fifth threshold, or if the first probability is less than or equal to the first threshold N3 consecutive times—then a smaller amount of data to be identified can be used for data collection, and the data to be identified can be collected at a lower first resolution. This reduces the workload of generating the amount of data to be identified, and compared to processing a larger amount of data, the power consumption and transmission bandwidth are significantly reduced when processing a smaller amount of data to be identified.

[0074] If a specific category of situation is detected in the detection environment, for example, if the number of times the first probability is greater than the first threshold within the fourth time period is greater than or equal to the sixth threshold, or if the first probability is greater than the first threshold for N4 consecutive times, then the resolution of the data to be identified needs to be increased, and the data to be identified is collected according to a higher second resolution; in this way, the recognition accuracy of the second-level recognition can be improved, thereby improving the corresponding service quality.

[0075] In this application, N3 and N4 can be any values, without restriction, and can be set according to the indicator requirements of the service scenario. The same applies to the fifth and sixth thresholds; they can also be any values ​​and can be set according to the indicator requirements of the service scenario.

[0076] In some embodiments, the method further includes: during the secondary identification process, if the number of times the first probability is less than or equal to the first threshold within a fifth time period is greater than or equal to a seventh threshold, or if the first probability is less than or equal to the first threshold for N5 consecutive times, the secondary identification is terminated; wherein, N5 is greater than 0.

[0077] In other words, if a specific type of situation disappears in the detection environment during the secondary recognition process (such as the disappearance of peeping events), the execution process can still be interrupted even if the secondary recognition program has not been completed, thereby saving power consumption.

[0078] Similarly, here, the first probability refers to the probability of belonging to a specific category, and that specific category refers to the target category detected in the corresponding application scenario.

[0079] In this application, N5 can be any value, without restriction, and can be set according to the indicator requirements of the service scenario. The same applies to the seventh threshold; it can also be any value and can be set according to the indicator requirements of the service scenario.

[0080] The main chip of a mobile terminal contains an Image Signal Processor (ISP), but the ISP in the main chip primarily handles traditional image algorithms, such as bad pixel correction, temporal noise reduction, 3D noise reduction, white balance, and automatic exposure. Figure 5 As shown, in addition to the main chip 501, there is also an image preprocessing chip 502 to perform differentiated algorithms, such as backlight photography, HDR photography, preview, and video effect enhancement in the RAW domain.

[0081] AON refers to functions such as air gestures and information anti-spying. AON solutions require low power consumption due to their persistent nature. Therefore, an image pre-processing chip is added to complete the first-level recognition; however, the performance of the pre-processing chip is lower than that of the AP chip, and its accuracy is still insufficient.

[0082] Based on this, the exemplary application of the embodiments of this application in a practical application scenario will be described below.

[0083] In this embodiment, real-time multi-level recognition is provided, which is superior in terms of accuracy and real-time performance. Multi-level recognition is performed on real-time image data, and the recognition accuracy is dynamically adjusted based on the results of the multi-level recognition. Power consumption is also adjusted by modifying the resolution and frame rate.

[0084] (1) Multiple people may appear in the field of view of the front-facing camera;

[0085] (2) The AON Sensor (an example of a data acquisition device) captures grayscale data of the image and makes the most basic motion judgment through the motion detection module of the AON Sensor; if there is a moving object, it triggers (3).

[0086] (3) The grayscale data of the image is sent to the pre-processing chip for first-level recognition. The first-level recognition is completed by the ISP and NPU of the pre-processing chip. The result output is: whether there is anyone else peeking at the phone and the corresponding confidence level. The recognized image is stored in the static random-access memory (SRAM) on the pre-processing chip. The SRAM is a ring buffer that can store a fixed number of grayscale images.

[0087] (4) When the confidence level of the first-level identification (i.e., the first probability that someone else is looking at the phone) > the threshold, the second-level identification of the AP is triggered;

[0088] (5) The AP sends the cached grayscale image to the preprocessing chip by notifying it;

[0089] (6) The AP completes high-precision secondary recognition based on the grayscale image transmitted from the preprocessing chip, and only transmits grayscale images with high confidence, and triggers corresponding actions.

[0090] The process of dynamically adjusting the AON recognition frequency (including the AON Sensor output resolution):

[0091] (1) As Figure 6 As shown, when the confidence level is less than the confidence threshold, the identification will be discarded directly without triggering any action response;

[0092] (2) When the confidence level > the confidence threshold, dynamic frame rate control is used. And as... Figure 6 As shown, the lower the confidence level, the higher the frame rate is used to ensure higher accuracy.

[0093] The real-time multi-level recognition provided in this application embodiment is superior to the current solution in terms of timeliness, accuracy, and real-time performance.

[0094] Furthermore, the recognition accuracy is dynamically adjusted based on the results of multi-level recognition, and the power consumption is adjusted by adjusting the resolution and frame rate, ensuring low power consumption while maintaining accuracy and real-time performance.

[0095] In some embodiments, AON achieves high accuracy and low power consumption by setting up primary recognition using low-resolution grayscale data and secondary recognition using high-resolution data. High accuracy is achieved through secondary recognition, which uses higher-resolution images and allows for the use of more precise AI models at the AP. Low power consumption is achieved by using primary recognition with low-resolution images under normal conditions, switching to high-resolution images only when the confidence level is sufficient.

[0096] In some embodiments, the triggering and exiting logic for secondary identification is as follows: When the confidence level of primary identification exceeds a threshold, secondary identification of the AP is triggered.

[0097] In some embodiments, the image for primary recognition is cached in the SRAM of the preprocessing chip. When secondary recognition is required, the cached image is transmitted to the AP for secondary recognition. Only grayscale images with high confidence are transmitted.

[0098] In some embodiments, the recognition accuracy is dynamically adjusted based on the results of multi-level recognition, and power consumption is adjusted by adjusting the resolution and frame rate.

[0099] In some embodiments, the AP starts the AP camera pipeline and configures the AON Sensor to output a high-resolution image, while the preprocessing chip performs first-level recognition simultaneously.

[0100] In some embodiments, if the confidence level of the first-level identification is low, the subsequent second-level identification is not performed. The system returns to the initial state, which achieves the goal of ending the second-level identification early and improving the real-time performance of the identification.

[0101] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps; or steps from different embodiments may be combined into a new technical solution.

[0102] Based on the foregoing embodiments, this application provides a data processing device, which includes various modules and units included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0103] Figure 7 This is a schematic diagram of the structure of the data processing device according to an embodiment of this application, as shown below. Figure 7 As shown, the device 700 includes:

[0104] The identification module 701 is used to perform first-level identification on the data to be identified to obtain a first identification result; and to perform the first-level identification on the next data to be identified if the first identification result does not meet the triggering condition for second-level identification.

[0105] The triggering module 702 is used to trigger the secondary recognition if the first recognition result meets the triggering condition. The secondary recognition is used to obtain a second recognition result based on at least one of the data to be recognized. The accuracy of the secondary recognition is higher than that of the first recognition.

[0106] In some embodiments, the first identification result includes a first probability of belonging to a specific category, and the triggering condition includes the first probability being greater than a first threshold; the secondary identification is used to obtain the second identification result based on the data to be identified in the data to be identified where the first probability is greater than or equal to a second threshold, and the second identification result includes a second probability of belonging to the specific category; wherein the second threshold is less than or equal to the first threshold.

[0107] In some embodiments, the data processing device 700 further includes a storage module and a data transmission module; wherein, the triggering module 702 is used to trigger the first processor to perform the secondary identification; the storage module is used to store the data to be identified and the corresponding first probability, or to store the data to be identified with the first probability greater than or equal to the second threshold; the data transmission module is used to send the stored data to be identified with the first probability greater than or equal to the second threshold to the first processor, so that the first processor performs the secondary identification based on the data to be identified with the first probability greater than or equal to the second threshold.

[0108] In some embodiments, the data processing apparatus 700 further includes an adjustment module, configured to: if the number of times the first probability is less than or equal to the first threshold within a first time period is greater than or equal to a third threshold, or if the first probability is less than or equal to the first threshold for N1 consecutive times, obtain a third probability based on at least one first probability less than or equal to the first threshold; adjust the current acquisition frequency of the data to be identified based on the third probability; wherein the third probability is positively correlated with the current acquisition frequency, and N1 is greater than 0.

[0109] In some embodiments, the adjustment module is further configured to: if the number of times the first probability is greater than the first threshold within a second time period is greater than or equal to a fourth threshold, or if the first probability is greater than the first threshold for N2 consecutive times, obtain a fourth probability based on at least one first probability greater than the first threshold; adjust the current collection frequency of the data to be identified based on the fourth probability; wherein the fourth probability is negatively correlated with the current collection frequency, and N2 is greater than 0.

[0110] In some embodiments, the adjustment module is further configured to: collect data to be identified at a first resolution if the number of times the first probability is less than or equal to the first threshold within a third time period is greater than or equal to a fifth threshold, or if the first probability is less than or equal to the first threshold for N3 consecutive times; collect data to be identified at a second resolution if the number of times the first probability is greater than the first threshold within a fourth time period is greater than or equal to a sixth threshold, or if the first probability is greater than the first threshold for N4 consecutive times; wherein N3 and N4 are both greater than 0, and the first resolution is less than the second resolution.

[0111] In some embodiments, the data processing device 700 further includes a control module, configured to terminate the secondary identification process if, during the secondary identification process, the number of times the first probability is less than or equal to the first threshold within a fifth time period is greater than or equal to a seventh threshold, or if the first probability is less than or equal to the first threshold for N5 consecutive times; wherein, N5 is greater than 0.

[0112] In some embodiments, the data to be identified includes image data, and the specific category includes the presence of voyeuristic events.

[0113] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0114] It should be noted that, in the embodiments of this application... Figure 7The module division of the data processing device shown is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or be integrated into one unit by two or more units. The integrated units can be implemented in hardware, as software functional units, or a combination of both.

[0115] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0116] This application provides an electronic device. Figure 8 This is a schematic diagram of the hardware entity of the electronic device according to an embodiment of this application, such as... Figure 8 As shown, the electronic device 800 includes a memory 801 and a processor 802. The memory 801 stores a computer program that can run on the processor 802. When the processor 802 executes the program, it implements the steps in the method provided in the above embodiments.

[0117] It should be noted that the memory 801 is configured to store instructions and applications executable by the processor 802, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data and video communication data) in the processor 802 and various modules in the electronic device 800. It can be implemented by flash memory or random access memory (RAM).

[0118] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0119] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0120] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0121] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0122] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0123] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0125] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0126] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0127] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0128] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0129] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0130] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0131] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0132] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: The data to be identified is subjected to primary identification to obtain the first identification result; If the first identification result does not meet the triggering condition for the second-level identification, the first-level identification is performed on the next data to be identified; the first identification result includes a first probability of belonging to a specific category, and the triggering condition includes the first probability being greater than a first threshold; If the first identification result meets the triggering condition, a second-level identification is triggered. The second-level identification is used to obtain a second identification result based on the data to be identified in the data to be identified where the first probability is greater than or equal to the second threshold. The identification accuracy of the second-level identification is higher than that of the first-level identification. The second identification result includes a second probability of belonging to the specific category. Wherein, the second threshold is less than the first threshold.

2. The method according to claim 1, characterized in that, The triggering of the secondary identification includes: triggering the first processor to perform the secondary identification; correspondingly, the method further includes: Store the data to be identified and the corresponding first probability, or store the data to be identified where the first probability is greater than or equal to the second threshold; The stored data to be identified, in which the first probability is greater than or equal to the second threshold, is sent to the first processor, so that the first processor can perform the secondary identification based on the data to be identified in which the first probability is greater than or equal to the second threshold.

3. The method according to claim 1, characterized in that, The method further includes: If the number of times the first probability is less than or equal to the first threshold within the first time period is greater than or equal to the third threshold, or if the first probability is less than or equal to the first threshold for N1 consecutive times, the third probability is obtained based on the first probability that is less than or equal to the first threshold at least once. Based on the third probability, the current acquisition frequency of the data to be identified is adjusted; wherein the third probability is positively correlated with the current acquisition frequency, and N1 is greater than 0.

4. The method according to claim 1, characterized in that, The method further includes: If the number of times the first probability is greater than the first threshold within the second time period is greater than or equal to the fourth threshold, or if the first probability is greater than the first threshold for N2 consecutive times, the fourth probability is obtained based on at least one first probability greater than the first threshold. Based on the fourth probability, the current acquisition frequency of the data to be identified is adjusted; wherein, the fourth probability is negatively correlated with the current acquisition frequency, and N2 is greater than 0.

5. The method according to claim 1, characterized in that, The method further includes: If the number of times the first probability is less than or equal to the first threshold within the third time period is greater than or equal to the fifth threshold, or if the first probability is less than or equal to the first threshold for N3 consecutive times, the data to be identified is collected according to the first resolution. If the number of times the first probability is greater than the first threshold within the fourth time period is greater than or equal to the sixth threshold, or if the first probability is greater than the first threshold for N4 consecutive times, the data to be identified is collected according to the second resolution; wherein N3 and N4 are both greater than 0, and the first resolution is less than the second resolution.

6. The method according to claim 1, characterized in that, The method further includes: During the secondary identification process, if the number of times the first probability is less than or equal to the first threshold within the fifth time period is greater than or equal to the seventh threshold, or if the first probability is less than or equal to the first threshold for N5 consecutive times, the secondary identification ends; wherein, N5 is greater than 0.

7. The method according to any one of claims 1 to 6, characterized in that, The data to be identified includes image data, and the specific category includes the presence of voyeuristic events.

8. A data processing apparatus, characterized in that, include: The recognition module is used to perform primary recognition on the data to be recognized and obtain the first recognition result; And for performing the first-level identification on the next data to be identified if the first identification result does not meet the triggering condition for second-level identification; the first identification result includes a first probability of belonging to a specific category, and the triggering condition includes the first probability being greater than a first threshold; A triggering module is used to trigger the secondary recognition if the first recognition result meets the triggering condition. The secondary recognition is used to obtain a second recognition result based on the data to be recognized in the data to be recognized where the first probability is greater than or equal to the second threshold. The accuracy of the secondary recognition is higher than that of the first recognition. The second identification result includes a second probability of belonging to the specific category; wherein the second threshold is less than the first threshold.

9. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images