A target recognition method, integrated processor, electronic device and storage medium

Abstract

This application provides a target recognition method, an integrated processor, an electronic device, and a storage medium, relating to the field of artificial intelligence technology. The integrated processor includes an algorithm processing subsystem, an input / output subsystem, and a power consumption control unit. The algorithm processing subsystem includes an NPU, a CPU, and a motion detection unit. The input / output subsystem includes a RAW image data processing interface. The method includes: the RAW image data processing interface writes RAW image frames to a buffer; the motion detection unit performs motion event detection on the RAW image frames, and in response to detecting a motion event, sends an interrupt signal to the power consumption control unit; the power consumption control unit, in response to receiving the interrupt signal, controls the CPU and NPU to enter a working state; the NPU identifies a moving target, feeds the identification result back to the CPU, and enters a sleep state; the CPU parses the identification result, outputs target identification information, and enters a sleep state. This reduces the power consumption of the device when performing target recognition.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a target recognition method, an integrated processor, an electronic device, and a storage medium. Background Technology

[0002] With the development of artificial intelligence (AI) technology, smart door locks, smart access control devices and other equipment are being used more and more in production and life scenarios. These devices can rely on their internal integrated processors to achieve target recognition.

[0003] In related technologies, the integrated processor in the aforementioned devices generally adopts a serial processing architecture of "image sensor → image signal processor (ISP) → AI chip". Specifically, the ISP is responsible for processing the high-resolution, high-frame-rate raw (RAW) image data acquired by the image sensor in the entire frame domain to obtain a standard format image; then, the AI ​​chip performs target recognition on the standard format image.

[0004] The image processing by the ISP requires a lot of computing resources, and the AI ​​chip needs to consume additional resources to perform target recognition on standard format images, resulting in high power consumption when the above devices perform target recognition. Summary of the Invention

[0005] The purpose of this application is to provide a target recognition method, an integrated processor, an electronic device, and a storage medium to reduce the power consumption of the electronic device when performing target recognition. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a target recognition method applied to an integrated processor in an electronic device. The integrated processor includes an algorithm processing subsystem, an input / output subsystem, and a power consumption control unit. The algorithm processing subsystem includes an NPU, a CPU, and a motion detection unit. The input / output subsystem includes a RAW image data processing interface connected to an external image sensor. The method includes:

[0007] The RAW image data processing interface extracts RAW image frames from the RAW image data acquired by the image sensor and writes the RAW image frames into a buffer.

[0008] The motion detection unit performs motion event detection on the RAW image frames in the buffer, and sends an interrupt signal to the power consumption control unit in response to the detection of a motion event.

[0009] The power consumption control unit responds to receiving an interrupt signal by controlling the CPU and the NPU to enter a working state.

[0010] The NPU identifies moving targets in RAW image frames based on the deployed set model, feeds the identification results back to the CPU, and then enters a sleep state.

[0011] The CPU parses the recognition result, outputs the target recognition information, and enters a sleep state.

[0012] Secondly, embodiments of this application provide an integrated processor, which includes an algorithm processing subsystem, an input / output subsystem, and a power consumption control unit. The algorithm processing subsystem includes an NPU, a CPU, and a motion detection unit. The input / output subsystem includes a RAW image data processing interface, which is connected to an external image sensor.

[0013] The RAW image data processing interface is used to extract RAW image frames from the RAW image data acquired by the image sensor and write the RAW image frames into a buffer.

[0014] The motion detection unit is used to detect motion events in the RAW image frames in the buffer, and in response to detecting a motion event, sends an interrupt signal to the power consumption control unit.

[0015] The power consumption control unit is used to control the CPU and the NPU to enter the working state in response to receiving an interrupt signal;

[0016] The NPU is used to identify moving targets in RAW image frames based on the deployed set model, feed the identification results back to the CPU, and then enter a sleep state.

[0017] The CPU is used to parse the recognition result, output target recognition information, and enter a sleep state.

[0018] Thirdly, embodiments of this application provide an electronic device, including:

[0019] Memory, used to store computer programs;

[0020] An integrated processor is used to implement the method described in the first aspect when executing a program stored in memory.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by an integrated processor, implements the method described in the first aspect.

[0022] Fifthly, embodiments of this application also provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in the first aspect.

[0023] As can be seen from the above, when applying the solution provided in this application for target recognition, on the one hand, the motion detection unit in the integrated processor directly performs motion event detection on the RAW image frames in the cache, rather than performing motion event detection on the converted standard format image; the NPU directly identifies moving targets in the RAW image frames based on the deployed set model, rather than identifying moving targets in the standard format image. That is, the integrated processor can perform motion detection and target recognition at the raw data level in the RAW domain, saving the computing resources required by the ISP to convert and reconstruct RAW image data, reducing the power consumption of the electronic device when performing target recognition while maintaining high-precision target recognition. On the other hand, the power consumption control unit in the integrated processor responds to the triggering of motion events, controlling the CPU and NPU to enter the working state, and the CPU and NPU will return to the sleep state after completing target recognition. In this way, when no motion event is triggered, the CPU and NPU are in a low-power state, and only enter the working state when a motion event is triggered, realizing the low-power working mechanism of "event-driven - on-demand activation - precise recycling", further reducing the power consumption of the electronic device when performing target recognition, which is especially suitable for battery-powered edge AI vision devices, maximizing the system energy efficiency ratio while ensuring functional integrity.

[0024] Furthermore, since no ISP is required to perform data conversion and reconstruction, the latency introduced by the ISP is eliminated, improving the target recognition performance of this solution in scenarios requiring rapid response. In addition, it overcomes the image quality degradation problem caused by the ISP in complex lighting environments and eliminates the loss of image detail information during the data conversion stage of the ISP, improving the performance and robustness of this solution for target recognition in various scenarios.

[0025] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0027] Figure 1 This is a schematic diagram of an integrated processor provided in an embodiment of this application;

[0028] Figure 2 A schematic diagram of a module interaction process provided in an embodiment of this application;

[0029] Figure 3 A flowchart illustrating a target recognition method provided in an embodiment of this application;

[0030] Figure 4 A schematic flowchart of a motion detection method provided in an embodiment of this application;

[0031] Figure 5 A signaling diagram of a module interaction flow provided in an embodiment of this application;

[0032] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0034] First, the implementing entity of the solution provided in the embodiments of this application will be described.

[0035] The implementation entity of the solution provided in this application embodiment is an integrated processor in an electronic device with target recognition function. The integrated processor includes at least an algorithm processing subsystem, an input / output subsystem, and a power consumption control unit. The electronic device with target recognition function can be called an intelligent vision device; the subsystems and units in the integrated processor are hardware modules, which can be simply referred to as modules. The functions of each module and the interaction process between modules are detailed in the following description.

[0036] The application scenarios of the solutions provided in the embodiments of this application will be introduced next.

[0037] The application scenario of the solution provided in this application embodiment is: the scenario where the integrated processor in an electronic device performs target recognition.

[0038] In related technologies, the integrated processor in the aforementioned devices generally adopts a serial processing architecture of "image sensor → image signal processor (ISP) → AI chip". Specifically, the ISP is responsible for performing full-frame-domain processing on the high-resolution, high-frame-rate raw (RAW) image data acquired by the image sensor to obtain a standard format image; then, the AI ​​chip performs target recognition on the standard format image based on the deployed AI model. The above solution has at least the following drawbacks:

[0039] 1. High power consumption bottleneck

[0040] Image processing by the ISP requires a lot of computing resources, and the AI ​​chip needs to consume even more resources to identify targets from standard format images. The dual computing process results in high system power consumption, which puts a lot of pressure on the power supply of electronic devices when they are identifying targets. This makes it difficult for electronic devices to achieve 24 / 7 "always online" monitoring.

[0041] 2. Significant processing delay

[0042] The ISP processing and AI chip inference form a serial pipeline. RAW image data must be reconstructed by the ISP before it can be analyzed by the AI ​​chip, which introduces a delay of hundreds of milliseconds. In scenarios such as capturing fast-moving targets and responding to sudden security incidents, it is easy to miss key information.

[0043] 3. Poor adaptability to extreme environments

[0044] Under complex lighting conditions such as low light, backlight, and strong glare, standard format images processed by the ISP are prone to overexposure / underexposure, loss of detail, and color distortion, which severely degrades the image quality input to the AI ​​model. This leads to a decrease in the target recognition performance of the AI ​​model (taking face detection as an example, actual data shows that this can cause a decrease in face recognition recall rate of more than 30%), and significantly reduces the robustness of the system.

[0045] 4. Redundancy in information links

[0046] Key information contained in RAW image data, such as high dynamic range, sensor noise characteristics, and weak light signals, is irreversibly compressed or filtered out during ISP conversion. This limits the AI ​​model's ability to deeply mine the original physical signals and affects the AI ​​model's target recognition performance.

[0047] In view of this, embodiments of this application provide a target recognition scheme to solve at least one of the above problems and reduce the power consumption of electronic devices when performing target recognition.

[0048] To facilitate understanding of the target recognition scheme provided in the embodiments of this application, the architecture of the integrated processor described above will be introduced first.

[0049] The aforementioned integrated processor includes at least an algorithm processing subsystem, an input / output subsystem, and a power consumption control unit. The algorithm processing subsystem includes an NPU, a CPU, and a motion detection unit. The input / output subsystem includes a RAW image data processing interface, which connects to an external image sensor.

[0050] The following section introduces the various modules in the integrated processor.

[0051] For the algorithm processing subsystem:

[0052] The NPU, as a dedicated hardware accelerator, is used to efficiently execute target recognition tasks based on deployed AI models, supporting parallel model computing and reducing power consumption; the CPU, as a general-purpose processing unit, is responsible for non-dedicated computing tasks such as task scheduling, data preprocessing / postprocessing, model loading, and system control logic; the motion detection unit is used to perform motion detection on the environment in which the electronic device is located.

[0053] For the input / output subsystem:

[0054] The RAW image data processing interface connects to an external image sensor for high-speed transmission of RAW image data acquired by the image sensor, serving as the front-end data channel of the vision system.

[0055] This application does not limit the specific type of the RAW image data processing interface described above; it can be any interface capable of high-speed transmission of RAW image data. Preferably, the RAW image data processing interface described above can be a Mobile Industry Processor Interface-Camera Serial Interface (MIPI-CSI).

[0056] In one possible implementation, the input / output subsystem may also include a storage control unit, which is connected to an external non-volatile memory and is used to manage the read and write operations of the non-volatile memory, and is responsible for loading NPU model parameters, system firmware and configuration files from the non-volatile memory.

[0057] Preferably, the aforementioned storage control unit may be a flash controller (FC), and the aforementioned non-volatile memory may be flash memory.

[0058] For power consumption control units:

[0059] The power control unit is responsible for the power management of the NPU and CPU. Specifically, it controls the power-on and power-off of the CPU and NPU, coordinates the wake-up and sleep processes of the CPU and NPU, and adjusts the operating voltage and clock frequency of the NPU. See the detailed explanation below.

[0060] In one possible implementation, the power control unit can be connected to an external power management integrated circuit (PMIC) to achieve power management for the NPU and CPU.

[0061] Preferably, the algorithm processing subsystem, input / output subsystem, and power consumption control unit can all be connected to the internal bus in the integrated processor, enabling high-speed data exchange between the modules.

[0062] Based on the above explanation, see Figure 1 , Figure 1 As an example of an integrated processor, based on Figure 1 This allows for an intuitive understanding of a specific structure of an integrated processor.

[0063] like Figure 2 As shown, after the integrated processor in the electronic device is powered on, the CPU and NPU are in a low-power state, which may be a power-off state or a deep sleep state; the power control system maintains a low-power standby state; the RAW image data processing interface in the output subsystem is activated and maintains a low-power listening state. The RAW image data processing interface extracts and caches RAW image frames, that is, it extracts RAW image frames from the RAW image data acquired by the external image sensor and caches the RAW image frames; it determines whether the motion detection unit has detected a motion event. If not, it continues to detect and returns to the judgment step. If yes, the power control unit controls the CPU and NPU to enter the working state, and the CPU and NPU enter the low-power state after completing the target recognition task.

[0064] The following section, using a flowchart, will provide a detailed explanation of the target recognition process performed by the integrated processor.

[0065] See Figure 3 This is a flowchart illustrating a target recognition method provided in an embodiment of this application. The method is applied to the integrated processor and includes the following steps S301-S305.

[0066] Step S301: The RAW image data processing interface extracts RAW image frames from the RAW image data acquired by the image sensor and writes the RAW image frames into the buffer.

[0067] The RAW image data processing interface is responsible for organizing RAW image data into RAW image frames and writing them into the cache unit in the integrated processor. The aforementioned cache unit can be a Double Data Rate Synchronous Dynamic Random-Access Memory (DDR SDRAM), etc., and this application embodiment does not limit it.

[0068] The following example illustrates the process of extracting RAW image frames, using MIPI-CSI as the RAW image data processing interface.

[0069] MIPI-CSI includes the MIPI Differential Physical Layer (D-PHY), MIPI Receive logic, and RAW image buffering logic. It extracts and buffers RAW image frames through the following steps A-C:

[0070] Step A: The MIPI D-PHY module uses one pair of source-synchronized differential clocks and four pairs of differential data lines for data transmission, converting the differential signal input from the image sensor into a serial digital signal.

[0071] Step B: The MIPI Receive logic completes frame synchronization of the serial data stream and extracts complete image frame data.

[0072] Specifically, the process first searches for the starting position of a frame of data in the serial data, and then, based on the starting position, performs alignment processing on the serial data stream and converts it into a complete RAW image frame.

[0073] Step C: The RAW image caching logic writes the extracted RAW image frames into DDR for caching via the internal bus according to the pre-allocated cache address space.

[0074] Step S302: The motion detection unit performs motion event detection on the RAW image frames in the buffer, and sends an interrupt signal to the power consumption control unit in response to the detection of a motion event.

[0075] It should be noted that in this step, the motion detection unit performs motion event detection directly on the RAW image frames, rather than on the converted standard format image.

[0076] The specific methods by which the motion detection unit detects motion events will be explained later and will not be detailed here.

[0077] Step S303: Upon receiving an interrupt signal, the power consumption control unit controls the CPU and NPU to enter the working state.

[0078] As explained above, the CPU and NPU in the algorithm subsystem may be in a power-off state or in a deep sleep state. Accordingly, the power control unit controls the CPU and NPU to enter the working state in different ways, which will be described separately below.

[0079] If the CPU and NPU are in deep sleep:

[0080] After receiving an interrupt signal, the power consumption control unit can wake up the CPU and NPU, release them from their reset state, and provide them with a working clock so that the CPU and NPU can return to their working state.

[0081] If the CPU and NPU are powered off:

[0082] After receiving an interrupt signal, the power consumption control unit can supply power to the CPU and NPU through an external PMIC, release them from their reset state, and provide them with a working clock so that the CPU and NPU can return to their working state.

[0083] Step S304: The NPU identifies moving targets in the RAW image frames based on the deployed configuration model, feeds back the identification results to the CPU, and enters sleep mode.

[0084] The NPU deployment model is configured as an AI model for target recognition, and this application embodiment does not limit its specific type and architecture. Furthermore, the specific type of target recognition task can be set according to specific task requirements, such as face recognition, workpiece recognition, etc., and this application embodiment does not limit this.

[0085] After the power consumption control unit controls the NPU to resume working state, the NPU can identify moving targets based on the set model, and enter a deep sleep state after feeding back the identification results, so as to enter the working state when the next motion event is detected.

[0086] Of course, before the NPU performs target recognition, the CPU can first perform preprocessing operations such as geometric transformation and size normalization on the RAW image frames.

[0087] In one possible implementation, in order to further reduce the power consumption of the electronic device when performing target recognition, the power control unit may also control the NPU and / or CPU to power down in response to the completion of the recognition task. After power-down, the model parameters loaded by the NPU will be cleared. Therefore, before performing this step, the CPU may instruct the storage control unit to read the model parameters from the non-volatile memory and instruct the NPU to load the model parameters.

[0088] In this scenario, after receiving an interrupt signal, the power control unit can first power the CPU and control it to enter the working state. After determining that the storage control unit has loaded the model parameters into the set cache space, the CPU can send an instruction to the power control unit requesting power to the NPU. In response to the above instruction, the power control unit will then power the NPU and control it to enter the working state. Then, the CPU can instruct the NPU to load the above model parameters.

[0089] Step S305: The CPU parses the recognition result, outputs the target recognition information, and enters sleep mode.

[0090] After obtaining the recognition results, the CPU can perform processing such as low-confidence detection box filtering and result integration on the recognition results, and output the final target recognition information.

[0091] As can be seen from the above, when applying the solution provided in this application for target recognition, on the one hand, the motion detection unit in the integrated processor directly performs motion event detection on the RAW image frames in the cache, rather than performing motion event detection on the converted standard format image; the NPU directly identifies moving targets in the RAW image frames based on the deployed set model, rather than identifying moving targets in the standard format image. That is, the integrated processor can perform motion detection and target recognition at the raw data level in the RAW domain, saving the computing resources required by the ISP to convert and reconstruct RAW image data, reducing the power consumption of the electronic device when performing target recognition while maintaining high-precision target recognition. On the other hand, the power consumption control unit in the integrated processor responds to the triggering of motion events, controlling the CPU and NPU to enter the working state, and the CPU and NPU will return to the sleep state after completing target recognition. In this way, when no motion event is triggered, the CPU and NPU are in a low-power state, and only enter the working state when a motion event is triggered, realizing the low-power working mechanism of "event-driven - on-demand activation - precise recycling", further reducing the power consumption of the electronic device when performing target recognition, which is especially suitable for battery-powered edge AI vision devices, maximizing the system energy efficiency ratio while ensuring functional integrity.

[0092] Furthermore, since no ISP is required to perform data conversion and reconstruction, the latency introduced by the ISP is eliminated, improving the target recognition performance of this solution in scenarios requiring rapid response. In addition, it overcomes the image quality degradation problem caused by the ISP in complex lighting environments and eliminates the loss of image detail information during the data conversion stage of the ISP, improving the performance and robustness of this solution for target recognition in various scenarios.

[0093] In one possible implementation, the above-mentioned model includes: a target detection model for detecting regions of interest in RAW image frames and a target recognition model for identifying targets within the regions of interest. In this case, the CPU can control the NPU to load model parameters in the following manner:

[0094] The storage control unit is instructed to read the first model parameters of the target detection model from the non-volatile memory, and the NPU is instructed to load the first model parameters; in response to the NPU successfully detecting the region of interest, the storage control unit is instructed to read the second model parameters of the target recognition model from the non-volatile memory, and the NPU is instructed to load the second model parameters.

[0095] As can be seen, the CPU first controls the NPU to load the first model parameters of the object detection model, so that the NPU can detect the region of interest based on the object detection model. After confirming that the NPU has successfully detected the region of interest, the CPU then controls the NPU to load the second model parameters of the object recognition model, so that the NPU can recognize the target based on the object recognition model. This avoids the waste of resources caused by loading the second model parameters when the region of interest has not been detected, and saves the device's storage bandwidth and power consumption.

[0096] To finely control the power consumption of the NPU and further save the device's power consumption, in one possible implementation, the CPU can send a first loading completion notification to the power control module in response to the NPU loading the first model parameters; the power control module, in response to receiving the first loading completion notification, adjusts the NPU's power consumption parameters based on the power consumption requirements of the target detection model during operation. And / or,

[0097] The CPU can send a second loading completion notification to the power control module in response to the NPU having loaded the second model parameters; upon receiving the second loading completion notification, the power control module adjusts the NPU's power parameters based on the power consumption requirements of the target recognition model during operation.

[0098] In other words, the power control unit can adaptively adjust the power consumption parameters of the NPU based on the power consumption requirements of different models deployed on the NPU during operation, so as to minimize power consumption while meeting the operational requirements of the models.

[0099] Preferably, in order to perform fine-grained power consumption control of the NPU in multiple aspects, the power consumption parameters mentioned above may include clock frequency and / or operating voltage.

[0100] For example, if the computing power requirement is high during the target detection model's operation phase, the power control unit can generate a higher frequency operating clock for the NPU and / or control the NPU to operate at a higher voltage; if the computing power requirement is low during the target recognition model's operation phase, the power control unit can generate a lower frequency operating clock for the NPU and / or control the NPU to operate at a lower voltage.

[0101] The following describes one method for motion event detection of RAW image frames in the buffer by the motion detection unit.

[0102] See Figure 4 This is a flowchart illustrating a motion event detection method provided in an embodiment of this application. In order to reduce the computational load required for motion detection and improve the efficiency of motion detection while ensuring the accuracy of motion detection, the motion detection unit can perform motion event detection using the following steps S401-S407 for each RAW image frame.

[0103] Step S401: Sample the RAW image frame by pixels to obtain a sampled frame.

[0104] In this step, sparse sampling can be performed on each row and each column of the RAW image frame. Preferably, only 1%-3% of the pixels of the RAW image frame can be retained as the sampling frame.

[0105] Specifically, a pseudo-random mode or a linear feedback shift register (LFSR) sequence generator can be used to sample pixels to ensure that pixels in each row and column are uniformly covered.

[0106] Step S402: Accumulate the pixel values ​​of each row of pixels in the sampling frame to obtain the accumulated pixel value of each row of pixels, and accumulate the pixel values ​​of each column of pixels in the sampling frame to obtain the accumulated pixel value of each column of pixels.

[0107] Specifically, the following parallel operation can be performed for each valid sampling point (x, y, pixel_value) in the sampling frame, where x and y represent the row number and column number, respectively, and pixel_value represents the pixel value:

[0108] Row projection accumulation: The pixel values ​​are accumulated into the row bucket current_row_hist[y] to obtain the pixel accumulation value of each row pixel.

[0109] Column projection accumulation: The pixel values ​​are synchronously accumulated to the column bucket current_col_hist[x] to obtain the pixel accumulation value of each column pixel.

[0110] Step S403: Determine the first difference between the pixel accumulation value of each row of pixels in the sampling frame and the pixel accumulation value of the same row of pixels in the reference frame, and use the accumulated first difference as the horizontal pixel variation value; determine the second difference between the pixel accumulation value of each column of pixels in the sampling frame and the pixel accumulation value of the same column of pixels in the reference frame, and use the accumulated second difference as the vertical pixel variation value.

[0111] The reference frame is updated based on RAW image frames without motion events; the specific update method will be explained later. The first reference frame can be obtained by sampling pixels from several initially acquired RAW image frames without motion events. The sampling method is the same as described above and will not be repeated here.

[0112] The horizontal and vertical pixel variations mentioned above can be referred to as the row projection difference and column projection difference between the sampled frame and the reference frame, respectively. Specifically, the row projection difference `row_diff` and column projection difference `col_diff` can be calculated using the following expressions:

[0113] row_diff=Σ|current_row_hist[i]-ref_row_hist[i]|, (i=0-H-1);

[0114] col_diff=Σ|current_col_hist[i]-ref_col_hist[i]|, (i=0-W-1).

[0115] Where current_row_hist[i] and current_col_hist[i] represent the pixel accumulation values ​​of the i-th row and i-th column of the sampled frame, respectively; ref_row_hist[i] and ref_col_hist[i] represent the pixel accumulation values ​​of the i-th row and i-th column of the reference frame, respectively; H represents the height of the sampled frame, i.e., the total number of rows; and W represents the width of the sampled frame, i.e., the total number of columns.

[0116] Step S404: Based on the horizontal pixel variation value and the vertical pixel variation value, determine the motion representation value that represents the probability of motion events in the RAW image frame.

[0117] Specifically, the motion score can be calculated using the following expression:

[0118] motion_score = α·row_diff + β·col_diff;

[0119] Here, α and β are the set directional weights, which can be set by staff based on experience and / or actual needs, such as 0.6 and 0.4 respectively.

[0120] Step S405: Determine whether the motion representation value is greater than the first threshold. If yes, proceed to step S406; if no, proceed to step S407.

[0121] Step S406: Determine that there is a motion event in the RAW image frame.

[0122] Step S407: Determine that there are no motion events in the RAW image frame.

[0123] In one possible implementation, the motion detection unit can send an interrupt signal to the power consumption control unit to eliminate transient interference if motion events are determined to exist in two consecutive RAW image frames.

[0124] In one possible implementation, the following steps may also be performed before executing step S403:

[0125] Accumulate the pixel accumulation value of each row of pixels in the sampled frame, and accumulate the pixel accumulation value of each column of pixels in the sampled frame. Based on the accumulation result, obtain the global brightness value of the sampled frame; determine the brightness difference between the global brightness value of the sampled frame and the global brightness value of the reference frame; in response to the brightness difference being greater than a second threshold, determine that there is no motion event in the RAW image frame; otherwise, perform the steps of determining the first difference and the second difference.

[0126] The aforementioned second threshold can be set by staff based on experience and / or actual needs, such as 15%.

[0127] The brightness difference Δ_brightness reflects the degree of difference between the global brightness value of the sampled frame and the global brightness value of the reference frame. If the difference is large, it can be determined that a sudden change in ambient light has occurred, and it can be directly determined that there is no motion event in the RAW image frame, without the need to perform subsequent steps; otherwise, it indicates that no sudden change in ambient light has occurred, and step S403 is executed to detect motion events based on motion characterization values.

[0128] In this way, when the ambient light change is determined by comparing the global brightness values, it can be directly determined that there is no motion event in the RAW image frame, eliminating the need for subsequent detection steps and further reducing the amount of computation required for motion detection.

[0129] The following section describes how the reference frame is updated.

[0130] If no motion events occur in any consecutive RAW image frames, the reference frame is updated as follows:

[0131] According to the first weight and the second weight, the pixel accumulation value of each row of pixels in the latest RAW image frame and the pixel accumulation value of the same row of pixels in the reference frame are weighted and calculated to be used as the pixel accumulation value of each row of pixels in the new reference frame.

[0132] According to the first weight and the second weight, the pixel cumulative value of each column of pixels in the latest RAW image frame and the pixel cumulative value of the same column of pixels in the reference frame are weighted and calculated to be the pixel cumulative value of each column of pixels in the new reference frame.

[0133] The second weight corresponding to the reference frame can be much greater than the first weight corresponding to the sampled frame. For example, the second weight can be 255 / 256 and the first weight can be 1 / 256. This means that when updating the reference frame, the content of the original reference frame is mainly considered, and the content of the current sampled frame is considered to a small extent. In other words, the current sampled frame is used to slowly update the existing reference frame.

[0134] Preferably, the reference frame can be updated using hardware shifting, as shown in the following expression:

[0135] ref_row_hist=(ref_row_hist×255+current_row_hist) >> 8;

[0136] ref_col_hist=(ref_col_hist×255+current_col_hist) >> 8.

[0137] Here, >> 8 represents a right shift operation of 8 bits, which is the equivalent operation of "dividing by 256".

[0138] In this way, if no motion events are detected for several consecutive frames, the current sampled frame can be used to slowly update the existing reference frame to adapt to the natural light drift changes in the environment. This ensures that the reference frame can be updated in real time according to the light changes, while also ensuring the stability of the reference frame.

[0139] To illustrate the interaction between modules in an integrated processor more intuitively, the following example, using a signaling diagram, illustrates a specific interaction process between modules.

[0140] See Figure 5 This application provides an interactive signaling diagram between modules, including the following steps S501-S519.

[0141] S501: The power consumption control unit responds to the received interrupt signal and controls the CPU to enter the working state.

[0142] Specifically, the power consumption control unit can power on the CPU, de-reset it, and generate the CPU's operating clock.

[0143] S502: The CPU controls the flash memory controller to load model parameters from external flash memory into memory.

[0144] S503: The CPU notifies that the power consumption control unit has been loaded.

[0145] S504: The power consumption control unit controls the NPU to enter the working state and adjusts the power consumption parameters of the NPU.

[0146] Specifically, the power control unit can power on the CPU, de-reset it, and generate the CPU's operating clock. Furthermore, the power control unit can adjust the NPU's voltage and clock frequency to match the power consumption requirements of the detection model during operation.

[0147] S505: The power consumption control unit notifies the NPU that power-on is complete.

[0148] S506: CPU controls NPU to load detection model parameters.

[0149] S507: NPU operation detection model.

[0150] S508: The NPU sends the detection results back to the CPU.

[0151] S509: The CPU notifies the power consumption control unit that the detection is complete.

[0152] S510: The power control unit adjusts the power consumption parameters of the NPU.

[0153] Similar to step S504, the power consumption control unit can adjust the voltage and clock frequency of the NPU to match the power consumption requirements of the recognition model during operation.

[0154] S511: The power consumption control unit notifies the CPU that the adjustment is complete.

[0155] S512: CPU controls NPU to load and identify model parameters.

[0156] S513: NPU runs the recognition model.

[0157] S514: The NPU feeds back the recognition results to the CPU.

[0158] S515: NPU enters sleep mode.

[0159] S516: The CPU notifies the power consumption control unit that recognition is complete.

[0160] S517: Power consumption control unit controls the NPU to power down.

[0161] S518: The power consumption control unit notifies the CPU that the NPU power-down is complete.

[0162] S519: CPU enters sleep mode.

[0163] Of course, as can be seen from the previous description, after step S519, the power consumption control unit can also control the CPU to power down to further save power consumption.

[0164] Therefore, the target recognition scheme provided in this application has at least the following advantages:

[0165] Activate on demand: The CPU and NPU are only activated when an event is triggered, avoiding power consumption during idle periods.

[0166] Phased loading: First, load the target detection model, and then decide whether to load the target recognition model based on the detection results, saving storage bandwidth and power consumption.

[0167] Highly efficient collaboration: The CPU is responsible for scheduling and decision-making, the NPU focuses on high-performance inference, and the power control unit manages power and timing in a unified manner.

[0168] Fast recovery: Supports millisecond-level wake-up and task switching, enabling an "always-on" edge intelligence experience.

[0169] Corresponding to the target recognition method described above, this application also provides an integrated processor.

[0170] The aforementioned integrated processor includes an algorithm processing subsystem, an input / output subsystem, and a power consumption control unit. The algorithm processing subsystem includes an NPU, a CPU, and a motion detection unit. The input / output subsystem includes a RAW image data processing interface, which connects to an external image sensor.

[0171] The RAW image data processing interface is used to extract RAW image frames from the RAW image data acquired by the image sensor and write the RAW image frames into a buffer.

[0172] The motion detection unit is used to detect motion events in the RAW image frames in the buffer, and in response to detecting a motion event, sends an interrupt signal to the power consumption control unit.

[0173] The power consumption control unit is used to control the CPU and the NPU to enter the working state in response to receiving an interrupt signal;

[0174] The NPU is used to identify moving targets in RAW image frames based on the deployed set model, feed the identification results back to the CPU, and then enter a sleep state.

[0175] The CPU is used to parse the recognition result, output target recognition information, and enter a sleep state.

[0176] As can be seen from the above, when applying the solution provided in this application for target recognition, on the one hand, the motion detection unit in the integrated processor directly performs motion event detection on the RAW image frames in the cache, rather than performing motion event detection on the converted standard format image; the NPU directly identifies moving targets in the RAW image frames based on the deployed set model, rather than identifying moving targets in the standard format image. That is, the integrated processor can perform motion detection and target recognition at the raw data level in the RAW domain, saving the computing resources required by the ISP to convert and reconstruct RAW image data, reducing the power consumption of the electronic device when performing target recognition while maintaining high-precision target recognition. On the other hand, the power consumption control unit in the integrated processor responds to the triggering of motion events, controlling the CPU and NPU to enter the working state, and the CPU and NPU will return to the sleep state after completing target recognition. In this way, when no motion event is triggered, the CPU and NPU are in a low-power state, and only enter the working state when a motion event is triggered, realizing the low-power working mechanism of "event-driven - on-demand activation - precise recycling", further reducing the power consumption of the electronic device when performing target recognition, which is especially suitable for battery-powered edge AI vision devices, maximizing the system energy efficiency ratio while ensuring functional integrity.

[0177] Furthermore, since no ISP is required to perform data conversion and reconstruction, the latency introduced by the ISP is eliminated, improving the target recognition performance of this solution in scenarios requiring rapid response. In addition, it overcomes the image quality degradation problem caused by the ISP in complex lighting environments and eliminates the loss of image detail information during the data conversion stage of the ISP, improving the performance and robustness of this solution for target recognition in various scenarios.

[0178] In one possible implementation, the input / output subsystem further includes a storage control unit connected to an external non-volatile memory.

[0179] The CPU is also used to, after entering the working state, instruct the storage control unit to read the model parameters from the non-volatile memory and instruct the NPU to load the model parameters;

[0180] The CPU is also used to send a notification that the identification is complete to the power consumption control unit after outputting the target identification information;

[0181] The power consumption control unit is also used to control the NPU and / or CPU to power down in response to receiving a notification that identification is complete.

[0182] This can further reduce the power consumption of electronic devices when performing target recognition.

[0183] In one possible implementation, the setting model includes: a target detection model for detecting regions of interest in RAW image frames and a target recognition model for identifying targets within regions of interest;

[0184] Specifically, the CPU is used to instruct the storage control unit to read the first model parameters of the target detection model from the non-volatile memory, and instruct the NPU to load the first model parameters; in response to the NPU successfully detecting the region of interest, the CPU instructs the storage control unit to read the second model parameters of the target recognition model from the non-volatile memory, and instructs the NPU to load the second model parameters.

[0185] As can be seen, the CPU first controls the NPU to load the first model parameters of the object detection model, so that the NPU can detect the region of interest based on the object detection model. After confirming that the NPU has successfully detected the region of interest, the CPU then controls the NPU to load the second model parameters of the object recognition model, so that the NPU can recognize the target based on the object recognition model. This avoids the waste of resources caused by loading the second model parameters when the region of interest has not been detected, and saves the device's storage bandwidth and power consumption.

[0186] In one possible implementation,

[0187] The CPU is also configured to send a first loading completion notification to the power control module in response to the NPU having loaded the first model parameters; the power control module, in response to receiving the first loading completion notification, adjusts the power consumption parameters of the NPU based on the power consumption requirements of the target detection model during operation.

[0188] And / or,

[0189] The CPU is also configured to send a second loading completion notification to the power control module in response to the NPU having loaded the second model parameters; the power control module, in response to receiving the second loading completion notification, adjusts the power consumption parameters of the NPU based on the power consumption requirements of the target recognition model during operation.

[0190] This allows for precise control of the NPU's power consumption, thereby further saving the device's power consumption.

[0191] In one possible implementation,

[0192] The power consumption parameters of the NPU include: clock frequency and / or operating voltage.

[0193] This allows for fine-grained power consumption control of the NPU in multiple aspects.

[0194] In one possible implementation, the motion detection unit is specifically used to perform motion event detection on each RAW image frame in the following manner:

[0195] Pixel sampling is performed on the RAW image frame to obtain the sampled frame;

[0196] The pixel values ​​of each row of pixels in the sampling frame are summed to obtain the pixel sum of each row of pixels, and the pixel values ​​of each column of pixels in the sampling frame are summed to obtain the pixel sum of each column of pixels.

[0197] A first difference is determined between the pixel accumulation value of each row of pixels in the sampled frame and the pixel accumulation value of the same row of pixels in the reference frame. The accumulated first difference is used as the horizontal pixel change value. A second difference is determined between the pixel accumulation value of each column of pixels in the sampled frame and the pixel accumulation value of the same column of pixels in the reference frame. The accumulated second difference is used as the vertical pixel change value. The reference frame is updated based on RAW image frames that do not have motion events.

[0198] Based on the horizontal and vertical pixel variation values, a motion representation value is determined that represents the probability of motion events existing in the RAW image frame.

[0199] If the motion representation value is greater than a first threshold, it is determined that there is a motion event in the RAW image frame; otherwise, it is determined that there is no motion event in the RAW image frame.

[0200] This allows for a reduction in the computational load required for motion detection and an improvement in its efficiency, while ensuring accuracy.

[0201] In one possible implementation, the motion detection unit is further configured to:

[0202] The pixel accumulation values ​​of each row of pixels in the sampling frame are accumulated, and the pixel accumulation values ​​of each column of pixels in the sampling frame are accumulated. The global brightness value of the sampling frame is obtained based on the accumulation results.

[0203] Determine the brightness difference between the global brightness value of the sampled frame and the global brightness value of the reference frame;

[0204] If the brightness difference is greater than a second threshold, it is determined that there is no motion event in the RAW image frame; otherwise, the steps of determining the first difference and the second difference are performed.

[0205] In this way, when the ambient light change is determined by comparing the global brightness values, it can be directly determined that there is no motion event in the RAW image frame, eliminating the need for subsequent detection steps and further reducing the amount of computation required for motion detection.

[0206] In one possible implementation, if no motion events occur in consecutive RAW image frames, the reference frame is updated as follows:

[0207] According to the first weight and the second weight, the pixel accumulation value of each row of pixels in the latest RAW image frame and the pixel accumulation value of the same row of pixels in the reference frame are weighted and calculated to be used as the pixel accumulation value of each row of pixels in the new reference frame.

[0208] According to the first weight and the second weight, the pixel cumulative value of each column of pixels in the latest RAW image frame and the pixel cumulative value of the same column of pixels in the reference frame are weighted and calculated to be the pixel cumulative value of each column of pixels in the new reference frame.

[0209] In this way, if no motion events are detected for several consecutive frames, the current sampled frame can be used to slowly update the existing reference frame to adapt to the natural light drift changes in the environment. This ensures that the reference frame can be updated in real time according to the light changes, while also ensuring the stability of the reference frame.

[0210] This application also provides an electronic device, such as... Figure 6 As shown, it includes:

[0211] Memory 601 is used to store computer programs;

[0212] The integrated processor 602 is used to implement the aforementioned target recognition method when executing the program stored in the memory 601.

[0213] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the integrated processor 602, communication interface, and memory 601 communicating with each other via the communication bus.

[0214] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0215] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0216] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned integrated processor.

[0217] The aforementioned integrated processor can be a general-purpose integrated processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0218] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by an integrated processor, implements the steps of the target recognition method described above.

[0219] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the target recognition methods described above.

[0220] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0221] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0222] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0223] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A target recognition method, characterized in that, An integrated processor for use in electronic devices, the integrated processor including an algorithm processing subsystem, an input / output subsystem, and a power consumption control unit, the algorithm processing subsystem including an NPU, a CPU, and a motion detection unit, the input / output subsystem including a RAW image data processing interface, the RAW image data processing interface being connected to an external image sensor, the method including: The RAW image data processing interface extracts RAW image frames from the RAW image data acquired by the image sensor and writes the RAW image frames into a buffer. The motion detection unit performs motion event detection on the RAW image frames in the buffer, and sends an interrupt signal to the power consumption control unit in response to the detection of a motion event. The power consumption control unit responds to receiving an interrupt signal by controlling the CPU and the NPU to enter a working state. The NPU identifies moving targets in RAW image frames based on the deployed set model, feeds the identification results back to the CPU, and then enters a sleep state. The CPU parses the recognition result, outputs the target recognition information, and enters a sleep state.

2. The method according to claim 1, characterized in that, The input / output subsystem further includes: a storage control unit, the storage control unit being connected to an external non-volatile memory; the method further includes: After the CPU enters the working state, it instructs the storage control unit to read the model parameters from the non-volatile memory and instructs the NPU to load the model parameters; After the CPU outputs the target identification information, it sends a notification that the identification is complete to the power consumption control unit. In response to receiving a notification that identification is complete, the power consumption control unit controls the NPU and / or CPU to power down.

3. The method according to claim 2, characterized in that, The defined model includes: a target detection model for detecting regions of interest in RAW image frames and a target recognition model for identifying targets within regions of interest; The CPU instructs the storage control unit to read model parameters from the non-volatile memory and instructs the NPU to load the model parameters, including: The storage control unit is instructed to read the first model parameters of the target detection model from the non-volatile memory, and the NPU is instructed to load the first model parameters; In response to the NPU successfully detecting the region of interest, the storage control unit is instructed to read the second model parameters of the target recognition model from the non-volatile memory, and the NPU is instructed to load the second model parameters.

4. The method according to claim 3, characterized in that, The method further includes: In response to the NPU having loaded the first model parameters, the CPU sends a first loading completion notification to the power control module; in response to receiving the first loading completion notification, the power control module adjusts the power consumption parameters of the NPU based on the power consumption requirements of the target detection model during operation. And / or, In response to the NPU having loaded the second model parameters, the CPU sends a second loading completion notification to the power control module; in response to receiving the second loading completion notification, the power control module adjusts the power consumption parameters of the NPU based on the power consumption requirements of the target recognition model during operation.

5. The method according to claim 4, characterized in that, The power consumption parameters of the NPU include: clock frequency and / or operating voltage.

6. The method according to any one of claims 1 to 5, characterized in that, The motion detection unit performs motion event detection on the RAW image frames in the buffer, including: For each RAW image frame, motion event detection is performed on that RAW image frame in the following manner: Pixel sampling is performed on the RAW image frame to obtain the sampled frame; The pixel values ​​of each row of pixels in the sampling frame are summed to obtain the pixel sum of each row of pixels, and the pixel values ​​of each column of pixels in the sampling frame are summed to obtain the pixel sum of each column of pixels. A first difference is determined between the pixel accumulation value of each row of pixels in the sampled frame and the pixel accumulation value of the same row of pixels in the reference frame. The accumulated first difference is used as the horizontal pixel change value. A second difference is determined between the pixel accumulation value of each column of pixels in the sampled frame and the pixel accumulation value of the same column of pixels in the reference frame. The accumulated second difference is used as the vertical pixel change value. The reference frame is updated based on RAW image frames that do not have motion events. Based on the horizontal and vertical pixel variation values, a motion representation value is determined that represents the probability of motion events existing in the RAW image frame. If the motion representation value is greater than a first threshold, it is determined that there is a motion event in the RAW image frame; otherwise, it is determined that there is no motion event in the RAW image frame.

7. The method according to claim 6, characterized in that, Before determining the first difference and the second difference, the process also includes: The pixel accumulation values ​​of each row of pixels in the sampling frame are accumulated, and the pixel accumulation values ​​of each column of pixels in the sampling frame are accumulated. The global brightness value of the sampling frame is obtained based on the accumulation results. Determine the brightness difference between the global brightness value of the sampled frame and the global brightness value of the reference frame; If the brightness difference is greater than a second threshold, it is determined that there is no motion event in the RAW image frame; otherwise, the steps of determining the first difference and the second difference are performed.

8. The method according to claim 6, characterized in that, If no motion events occur in any consecutive RAW image frames, the reference frame is updated as follows: According to the first weight and the second weight, the pixel accumulation value of each row of pixels in the latest RAW image frame and the pixel accumulation value of the same row of pixels in the reference frame are weighted and calculated to be used as the pixel accumulation value of each row of pixels in the new reference frame. According to the first weight and the second weight, the pixel cumulative value of each column of pixels in the latest RAW image frame and the pixel cumulative value of the same column of pixels in the reference frame are weighted and calculated to be the pixel cumulative value of each column of pixels in the new reference frame.

9. An integrated processor, characterized in that, The integrated processor includes an algorithm processing subsystem, an input / output subsystem, and a power consumption control unit. The algorithm processing subsystem includes an NPU, a CPU, and a motion detection unit. The input / output subsystem includes a RAW image data processing interface, which connects to an external image sensor. The RAW image data processing interface is used to extract RAW image frames from the RAW image data acquired by the image sensor and write the RAW image frames into a buffer. The motion detection unit is used to detect motion events in the RAW image frames in the buffer, and in response to detecting a motion event, sends an interrupt signal to the power consumption control unit. The power consumption control unit is used to control the CPU and the NPU to enter the working state in response to receiving an interrupt signal; The NPU is used to identify moving targets in RAW image frames based on the deployed set model, feed the identification results back to the CPU, and then enter a sleep state. The CPU is used to parse the recognition result, output target recognition information, and enter a sleep state.

10. An electronic device, characterized in that, It includes an integrated processor, a communication interface, a memory, and a communication bus, wherein the integrated processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; An integrated processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by an integrated processor, implements the steps of the method described in any one of claims 1-8.

Images