Monitor system and control method thereof, computer equipment and storage medium

Abstract

The invention provides a broadcast monitoring system and a control method thereof, and a storage medium, and the broadcast monitoring system is integrated with a neural network model. The neural network model comprises a plurality of network layers connected in sequence and a loss function algorithm; the broadcast monitoring system at least comprises a storage and calculation integrated subsystem which is in communication connection with the terminal. The storage and calculation integrated subsystem comprises a processing module, a storage and calculation integrated module and a storage module; a plurality of sequentially connected network layers are integrated in the storage and calculation integrated module, and a loss function algorithm is integrated in the processing module; the storage and calculation integrated module is configured to read the target image from the storage module, and the target image is processed by using a plurality of sequentially connected network layers to obtain a target output result; the processing module is configured to read the target output result and process the target output result by using a loss function algorithm to obtain a sensing detection result; and under the condition that the perception detection result meets a preset condition, sending the perception detection result to the terminal, so that the terminal determines a display state based on the perception detection result.

Description

Monitoring system and control method thereof, computer equipment and storage medium Technical Field

[0001] The present disclosure belongs to the technical field of image recognition and terminal monitoring, and particularly relates to a monitoring system and a control method thereof, a computer device, and a storage medium. Background Art

[0002] Traditional portable terminals can be used as lightweight applications in front of the system-on-chip (SOC), taking on external environment perception tasks such as human presence detection, voice detection, and gaze detection. Smart display terminals use the SOC's self-perception results to determine the current display state. However, portable terminals often suffer from "battery anxiety" issues. For real-time detection in perception task application scenarios, when using the SOC system for processing, large amounts of data are frequently transferred between the storage and computing units, resulting in 60% to 90% of energy consumption being used for data transfer rather than data calculation. This is also the main cause of the SOC system's energy consumption wall problem.

[0003] With the development of chip technology, the minimum process technology has become difficult to continue to decline, and the von Neumann architecture has gradually become difficult to continue Moore's Law. People are eager to find an emerging computing method or architecture to replace the traditional chip architecture.

[0004] Summary of the Invention

[0005] The present disclosure aims to provide a monitoring system and a control method thereof, a computer device and a storage medium.

[0006] In a first aspect, the technical solution adopted to solve the technical problem of the present disclosure is a monitoring system that integrates a neural network model; the neural network model includes multiple sequentially connected network layers and a loss function algorithm; the monitoring system includes at least a storage and computing subsystem, the storage and computing subsystem being communicatively connected to a terminal; the storage and computing subsystem includes a processing module, a storage and computing module, and a storage module; the multiple sequentially connected network layers are integrated in the storage and computing module, and the loss function algorithm is integrated in the processing module;

[0007] The storage and computing module is configured to read the target image from the storage module and process it using multiple sequentially connected network layers to obtain a target output result;

[0008] The processing module is configured to read the target output result and process it using a loss function algorithm to obtain a perception detection result; if the perception detection result meets a preset condition, the perception detection result is sent to the terminal so that the terminal determines the display state based on the perception detection result.

[0009] In some embodiments, the storage and computing module includes an instruction cache unit, a data cache unit, and a storage and computing unit; the storage and computing unit is integrated with a multi-layer sequentially connected network layer;

[0010] The storage and calculation unit is configured to respond to the first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, use the first layer of the network layer to perform weighted processing on the image data group according to multiple pre-stored weight data groups, obtain multiple intermediate sub-results, and store them in the data cache unit; continue to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeat until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit; use the multiple groups of channel data as the target image to be detected in the next layer of the network layer, execute the process of sequentially reading the image data group of the target image from the data cache unit, and determine the multiple groups of channel data until the target output result of the target image is output after passing through each of the network layers in sequence.

[0011] In some embodiments, the storage and computing unit is specifically configured to respond to a first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, and store them in the data cache unit; respond to a second instruction cached in the instruction cache unit, read the image data group of the target image from the data cache unit; respond to a third instruction cached in the instruction cache unit, use the first layer of the network layer to perform weighted processing on the image data group according to multiple sets of pre-stored weight data groups, obtain multiple intermediate sub-results, and store them in the data cache unit; continue to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeat until the intermediate sub-results corresponding to all image data groups of the target image are output, and multiple sets of channel data are obtained and stored in the data cache unit; use the multiple sets of channel data as the target image to be detected in the next layer of the network layer, execute the process of sequentially reading the image data group of the target image from the data cache unit, and determine the multiple sets of channel data until the target output result of the target image is output after passing through each of the network layers in sequence.

[0012] In some embodiments, a set of the image data includes n eigenvalues, and a set of the weight data includes n weight values; the i-th eigenvalue corresponds to the i-th weight value; n and i are both positive integers, and i is less than or equal to n;

[0013] The storage and calculation unit is specifically configured to respond to the first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, and store it in the data cache unit; respond to the second instruction cached in the instruction cache unit, sequentially read the n eigenvalues ​​in the image data group from the data cache unit; respond to the third instruction cached in the instruction cache unit, use the first layer of the network layer, for each of the multiple weight data groups, weight the i-th eigenvalue and the i-th weight value in the weight data group to obtain the The intermediate sub-result is obtained and stored in the data cache unit; the process of sequentially reading the next image data group of the target image from the storage module is continued, and repeated until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit; the multiple groups of channel data are used as the target image to be detected in the next layer of the network layer, and the image data group of the target image is sequentially read from the data cache unit, and the process of determining multiple groups of channel data is performed until the target output result of the target image is output after passing through each of the network layers in sequence.

[0014] In some embodiments, the amount of data in the image data set is greater than the maximum amount of data that can be retrieved by the storage and computing unit;

[0015] The storage and calculation unit is specifically configured to respond to the first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, and store it in the data cache unit; respond to the second instruction cached in the instruction cache unit, read the first data subset in the image data group from the data cache unit; the data amount of the first data subset is the maximum data amount; respond to the third instruction cached in the instruction cache unit, use the first layer of the network layer to weight the first part of the weight data of each of the multiple groups of weight data groups stored in advance, and obtain multiple intermediate partial results; read the second data subset in the image data group from the data cache unit, the data of the second data subset is the remaining image data in the image data group except the first data subset; multiple intermediate partial results are weighted. The molecular results are respectively used as the initial values ​​when the second data subset is weighted by the multiple groups of weight data groups, and the second data subset is weighted according to the multiple intermediate partial molecular results and the second part of the weight data of each of the multiple groups of weight data groups stored in advance to obtain multiple intermediate sub-results, and store them in the data cache unit; continue to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeat until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit; use the multiple groups of channel data as the target image to be detected in the next layer of the network layer, execute the process of sequentially reading the image data group of the target image from the data cache unit, and determine the multiple groups of channel data until the target output result of the target image is output after passing through each of the network layers in sequence.

[0016] In some embodiments, a set of the image data includes n eigenvalues, and a set of the weight data includes n weight values; the i-th eigenvalue corresponds to the i-th weight value; n and i are both positive integers, and i is less than or equal to n;

[0017] The storage and calculation unit is specifically configured to respond to the first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, and store it in the data cache unit; respond to the second instruction cached in the instruction cache unit, sequentially read n1 eigenvalues ​​in the first data subset from the data cache unit; respond to the third instruction cached in the instruction cache unit, use the first layer of the network layer, for the first part of the weight data of each of the multiple groups of weight data groups, weight the i1th eigenvalue and the i1th weight value in the weight data group to obtain the intermediate sub-result; n1 and i1 are both positive integers, and i1 is less than or equal to n1; sequentially read n2 eigenvalues ​​in the second data subset from the data cache unit, n2+n1=n; and respectively store the multiple intermediate sub-results As the initial value when multiple groups of weight data groups are weighted for the second data subset, for the second part of the weight data of each of the multiple groups of weight data groups, the intermediate part of the subset result as the initial value, the i2th eigenvalue and the i2th weight value in the weight data group are weighted to obtain the intermediate sub-result, and stored in the data cache unit; n2 and i2 are both positive integers, and i2 is less than or equal to n2; continue to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeat until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit; use the multiple groups of channel data as the target image to be detected in the next layer of the network layer, execute the process of sequentially reading the image data group of the target image from the data cache unit, and determine the multiple groups of channel data, until the target output result of the target image is output after passing through each of the network layers in sequence.

[0018] In some embodiments, the neural network model further includes a normalization function algorithm and an activation function algorithm; the normalization function algorithm and the activation function algorithm are both integrated in the processing module;

[0019] The storage and calculation unit is configured to respond to the first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, use the first layer of the network layer to perform weighted processing on the image data group according to multiple pre-stored weight data groups, obtain multiple intermediate sub-results, and store them in the data cache unit; continue to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeat until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit;

[0020] The processing module is configured to read multiple sets of channel data, perform normalization processing using the normalization function algorithm, and obtain a normalization result; process the normalization result using the activation function algorithm to obtain an activation result, and store it in the data cache unit; the activation result is the target image to be detected in the next layer of the network layer;

[0021] The storage and computing unit is configured to respond to the fourth instruction cached in the instruction cache unit, use the next layer of the network layer to execute the process of sequentially reading the image data group of the target image from the data cache unit, and determine multiple groups of channel data until the target output result of the target image is output after passing through each of the network layers in sequence.

[0022] In some embodiments, the processing module is further configured to obtain the image to be detected captured by the sensing device at a preset time interval; divide the image to be detected into multiple sub-images according to the device information of the sensing device, and determine the order to be detected of each sub-image; store the multiple sub-images corresponding to the image to be detected in the storage module; and read the sub-images from the storage module in sequence, adjust the size of the sub-image according to a preset ratio, obtain the target image corresponding to the sub-image, and store the target image in the storage module; the order to be detected of each sub-image is the same as the order to be detected of each target image.

[0023] In some embodiments, the processing module divides the image to be detected into multiple sub-images based on the device information of the sensing device. Specifically, based on the device information of the sensing device, the image to be detected is divided into the 1st to mth sub-images, and the i+1th sub-image covers the i-th sub-image, and the center point of each sub-image is located in the central area of ​​the image to be detected; the order of detection of the 1st to mth sub-images is 1 to m.

[0024] In some embodiments, the processing module is further configured to, when the perception detection result corresponding to the target image does not meet the preset conditions, sequentially traverse the other target images remaining in the images to be detected except the target image that has been detected in accordance with the order to be detected, until the perception detection result meets the preset conditions, and then send the perception detection result to the terminal; the other target images are the target images remaining in the multiple target images corresponding to the image to be detected except the target images in the historical traversal process; determine whether all the target images corresponding to the image to be detected have been traversed, and when all the target images corresponding to the image to be detected have been traversed, send the perception detection result corresponding to the last traversed target image to the terminal.

[0025] In a second aspect, embodiments of the present disclosure further provide a monitoring system, wherein a neural network model is integrated; the neural network model includes multiple sequentially connected network layers and a loss function algorithm; the monitoring system includes a storage and computing subsystem and a terminal, the storage and computing subsystem being communicatively connected to the terminal; the storage and computing subsystem includes a processing module, a storage and computing module, and a storage module; the multiple sequentially connected network layers are integrated in the storage and computing module, and the loss function algorithm is integrated in the processing module;

[0026] The storage and computing module is configured to read the target image from the storage module and process it using multiple sequentially connected network layers to obtain a target output result;

[0027] The processing module is configured to read the target output result and process it using a loss function algorithm to obtain a perception detection result; if the perception detection result meets a preset condition, send the perception detection result to the terminal;

[0028] The terminal is configured to determine a display state based on the perception detection result.

[0029] In some embodiments, the terminal further includes a display module and a main control module;

[0030] The main control module is configured to, if it is determined that the perception detection result meets the preset conditions, reset the sleep timer to receive the perception detection result sent by the storage and computing subsystem by responding to the interrupt request; and send a wake-up request to the display module; if it is determined that the perception detection result does not meet the preset conditions and the time between the current system time and the last time the display module was woken up is greater than a preset time length, send a sleep request to the display module and enter a sleep state itself;

[0031] The display module is configured to respond to the wake-up request and determine that the display state is normal display; and respond to the sleep request and determine that the display state is sleep.

[0032] In a third aspect, an embodiment of the present disclosure further provides a control method for a monitoring system, wherein the monitoring system is integrated with a neural network model; the neural network model includes multiple sequentially connected network layers and a loss function algorithm; the monitoring system includes at least a storage and computing subsystem, the storage and computing subsystem being communicatively connected to a terminal; the storage and computing subsystem includes a processing module, a storage and computing module, and a storage module; the multiple sequentially connected network layers are integrated in the storage and computing module, and the loss function algorithm is integrated in the processing module;

[0033] The storage and computing module reads the target image from the storage module and processes it using multiple sequentially connected network layers to obtain a target output result;

[0034] The processing module reads the target output result and processes it using a loss function algorithm to obtain a perception detection result; if the perception detection result meets a preset condition, the perception detection result is sent to the terminal so that the terminal determines a display state based on the perception detection result.

[0035] In a fourth aspect, an embodiment of the present disclosure further provides a control method for a monitoring system, wherein the monitoring system is integrated with a neural network model; the neural network model includes multiple sequentially connected network layers and a loss function algorithm; the monitoring system includes a storage and computing subsystem and a terminal, the storage and computing subsystem being communicatively connected to the terminal; the storage and computing subsystem includes a processing module, a storage and computing module, and a storage module; the multiple sequentially connected network layers are integrated in the storage and computing module, and the loss function algorithm is integrated in the processing module;

[0036] The storage and computing module reads the target image from the storage module and processes it using multiple sequentially connected network layers to obtain a target output result;

[0037] The processing module reads the target output result and processes it using a loss function algorithm to obtain a perception detection result; if the perception detection result meets a preset condition, the perception detection result is sent to the terminal;

[0038] The terminal determines a display state based on the perception detection result.

[0039] In the fifth aspect, an embodiment of the present disclosure further provides a computer non-volatile readable storage medium, wherein a computer program is stored on the computer non-volatile readable storage medium, and when the computer program is executed by a processor, the steps of the control method of the monitoring system as described in the third aspect and / or the fourth aspect are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] FIG1 is a schematic structural diagram of a monitoring system provided by an embodiment of the present disclosure;

[0041] FIG2 is a schematic diagram of the specific structure of a storage and computing integrated subsystem provided by an embodiment of the present disclosure;

[0042] FIG3 is a schematic diagram of the architecture of a neural network model provided by an embodiment of the present disclosure;

[0043] FIG4a and FIG4b are schematic diagrams of feature weighting provided in an embodiment of the present disclosure;

[0044] FIG5 is a schematic diagram of data interaction between a storage and computing integrated module according to an embodiment of the present disclosure;

[0045] FIG6 a is a schematic diagram of a storage-computation-in-one circuit provided in an embodiment of the present disclosure;

[0046] FIG6 b is a schematic diagram of feature weighted shifting provided by an embodiment of the present disclosure;

[0047] FIG7 is a schematic diagram of convolution deployment allocation provided by an embodiment of the present disclosure;

[0048] FIG8 is a schematic diagram of an image block preprocessing method provided by an embodiment of the present disclosure;

[0049] FIG9 is a schematic structural diagram of another monitoring system provided by an embodiment of the present disclosure;

[0050] FIG10 is a schematic diagram of another monitoring system provided by an embodiment of the present disclosure;

[0051] FIG11a is a diagram of an exemplary terminal workflow provided by an embodiment of the present disclosure;

[0052] FIG11b is a flowchart of an exemplary storage-computing subsystem workflow provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0054] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The words "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "an" or "the" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0055] In this disclosure, "multiple or several" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0056] To address the memory wall problem, the industry has adopted numerous architectural design optimization solutions, such as multi-level caches and highly complex and sophisticated dynamic real-time data flow control and management. However, these solutions are theoretically incapable of being automatically "understood" by compilers and unable to automatically optimize the chip's executable code. Consequently, manual optimization requires significant investment.

[0057] In a first aspect, the present disclosure provides a monitoring system integrated with a neural network model; the neural network model includes multiple layers of sequentially connected network layers and a loss function algorithm; the monitoring system includes at least a storage and computing subsystem, which is communicatively connected to a terminal; the storage and computing subsystem includes a processing module, a storage and computing module and a storage module; the multiple layers of sequentially connected network layers are integrated in the storage and computing module, and the loss function algorithm is integrated in the processing module; the storage and computing module is configured to read a target image from the storage module, and process it using the multiple layers of sequentially connected network layers to obtain a target output result; the processing module is configured to read the target output result, and process it using the loss function algorithm to obtain a gaze detection result; when the gaze detection result indicates that someone is staring at the screen, the gaze detection result is sent to the terminal, so that the terminal determines the display status based on the gaze detection result.

[0058] The present disclosure utilizes a storage-computing subsystem that integrates multiple sequentially connected network layers into the storage-computing module. This integrated storage and computing module, a system architecture that integrates storage and computing, supports data operations (such as matrix operations) at the network layer, significantly avoiding energy loss during data transfer and reducing system power consumption. Furthermore, the monitoring system provided by the present disclosure integrates a neural network model that predicts perception tasks. Compared to traditional recognition technologies, this improves the detection accuracy of perception tasks, allowing for more accurate control of when a terminal enters sleep or actively wakes up, further reducing system power consumption.

[0059] Let's briefly introduce the perception detection tasks involved in this disclosure, which are generally applied to various smart display terminals. They can be roughly divided into two categories: one is a large-scale display terminal that needs to be viewed from a distance, and the other is a desktop display, which is usually within 1 meter.

[0060] For ease of understanding, a monitoring system provided in an embodiment of the present disclosure is first described in detail. The monitoring system provided in the present disclosure can be understood as a system that controls the playback or sleep timing of a terminal by detecting a perception task.

[0061] FIG1 is a schematic structural diagram of a monitoring system provided by an embodiment of the present disclosure. As shown in FIG1 , the monitoring system includes at least a storage and computing subsystem 10, which is communicatively connected to the terminal; the storage and computing subsystem 10 includes a processing module 1, a storage and computing module 2, and a storage module 3. A neural network model is integrated in the monitoring system for performing perception task detection to control the display status of the terminal. The neural network model includes a multi-layer sequentially connected network layer 201 and a loss function algorithm 11, wherein the more complex loss function algorithm 11 in the neural network model can be integrated into the processing module 1 of the storage and computing subsystem 10, and the multi-layer sequentially connected network layer 201 in the neural network model can be integrated into the storage and computing module 2 of the storage and computing subsystem 10. The storage and computing subsystem 10 also includes a bus 4, and the processing module 1 can be electrically connected to the storage and computing module 2 and the storage module 3 through the bus 4 to realize data transmission. The storage and computing module 2 and the storage module 3 can be electrically connected through an interface protocol to realize data transmission.

[0062] Exemplarily, the processing module 1 in the storage-computing subsystem 10 is, for example, the main processor CPU of the storage-computing subsystem 10; the storage module 3 in the storage-computing subsystem 10 is, for example, the external memory DRAM in the storage-computing subsystem 10.

[0063] The storage and computing module 2 is configured to read the target image from the storage module 3 and process it using the multi-layer sequentially connected network layer 201 to obtain the target output result. Here, the target output result can be stored in the storage module 3; alternatively, the target output result can be sent directly to the processing module 1. The present embodiment uses the example of storing the target output result in the storage module 3 as an example.

[0064] The storage-computing module 2 integrates multiple sequentially connected network layers 201. The network layer 201 performs matrix operations and caches the calculation results to facilitate subsequent calls to the next network layer 201. For example, the next network layer 201 only needs to read the calculation data from the data cache unit 22 of the storage-computing module 2 itself, improving data scheduling efficiency. Compared with the memory wall problem in the prior art, the present disclosure uses the storage-computing module 2 to avoid energy loss caused by the data transfer process, thereby improving calculation efficiency and reducing system power consumption. It has a high application prospect for portable devices.

[0065] The processing module 1 is configured to read the target output result and process it using the loss function algorithm 11 to obtain a perception detection result; when the perception detection result meets the preset conditions, the perception detection result is sent to the terminal so that the terminal determines the display state based on the perception detection result.

[0066] The storage and computing subsystem 10 uses the storage module 3 to store the target image, the storage and computing module 2 to perform pre-processing for perception detection, and the processing module 1 to perform post-processing for perception detection. For example, the loss function algorithm 11 is used to process the target output result of the last network layer 201 to determine the perception detection result of the target image under the perception task.

[0067] The preset condition can be a pre-set condition indicating the presence of perception corresponding to the perception detection task. For example, for a gaze detection task, the preset condition can be that someone is staring at the screen; or, for a human figure detection task, the preset condition can be that a target object (person) exists; or, for a voice detection task, the preset condition can be that a sound is monitored, and so on.

[0068] Taking the gaze detection task as an example, when the perception detection result indicates that someone is gazing at the screen, the terminal determines that its current display state is normal.

[0069] In some embodiments, the storage-computing module 2 includes at least an instruction cache unit 21, a data cache unit 22, and a storage-computing unit 23; the storage-computing unit 23 is integrated with a multi-layer sequentially connected network layer 201. The data cache unit 22 is a cache integrated in the storage-computing module 2. There is no memory wall between the storage-computing unit 23 and the data cache unit 22, which facilitates the storage-computing unit 23 to read data from the data cache unit 22. Figure 2 is a schematic diagram of the specific structure of a storage-computing subsystem provided by an embodiment of the present disclosure, as shown in Figure 2, where I-Cache represents the instruction cache unit 21, D-Cache represents the data cache unit 22, and PIM-Macho represents the storage-computing unit 23.

[0070] FIG3 is a schematic diagram of the architecture of a neural network model provided by an embodiment of the present disclosure. As shown in FIG3 , taking a convolutional neural network model as an example, the multiple layers of sequentially connected network layers 201 may include, for example, multiple layers of convolutional layers conv and fully connected layers fc connected to the multiple layers of sequentially connected convolutional layers. For example, the neural network model includes ten layers of convolutional layers conv and one fully connected layer fc.

[0071] The storage and calculation unit 23 is configured to respond to the first instruction cached in the instruction cache unit 21, read the image data group of the target image from the storage module 3 in sequence, and use the first network layer 201 to perform weighted processing on the image data group according to multiple sets of pre-stored weight data groups to obtain multiple intermediate sub-results, and store them in the data cache unit 22.

[0072] Here, the first instruction may be, for example, an instruction pre-cached by the instruction cache unit 21. Exemplarily, the processing module 1 is configured to read out the first instruction from the instruction cache unit 21 and parse it after storing the target image in the storage module 3; the storage and calculation unit 23 executes the subsequent process in response to the first instruction parsed by the processing module 1. When the storage and calculation unit 23 is executed, it may instruct the processing module 1 to perform other operations by way of an interrupt. The "other operations" here may be, for example, the operation of the subsequent processing module 1 reading the second instruction, or the process of the processing module 1 performing some intermediate calculations internally and then notifying the storage and calculation unit 23 to execute subsequent instructions. One set of image data groups [a1 a2 ... an] includes image data of multiple target images, and one set of image data groups contains only part of the image data of the target image. A set of weight data groups [b11 b12 … b1n] represents p convolution kernels. The number of convolution kernels p is the same as the number of channels of the target image. A convolution kernel includes multiple pre-set weight values. For example, a 2×2 convolution kernel includes 4 weight values. If a set of weight data groups includes 3 2×2 convolution kernels, then a set of weight data groups includes 12 weight values. Multiple sets of weight data groups are represented using matrices as follows:

[0073] Among them, b1n represents the nth weight value in the first group of weight data; bqn represents the nth weight value in the qth group of weight data.

[0074] Figures 4a and 4b are schematic diagrams of feature weighting provided by the embodiments of the present disclosure, respectively. As shown in Figures 4a and 4b, when the image data group is weighted using the above-mentioned q groups of weight data groups, it can be understood as performing a matrix operation, and the number of image data in one group of image data groups is the same as the number of weight values ​​in one group of weight data groups, and they correspond one to one. When the image data group is weighted according to multiple groups of pre-stored weight data groups, one group of image data groups is weighted with one group of weight data groups to obtain an intermediate sub-result; one group of image data groups is weighted with multiple groups of weight data groups to obtain multiple intermediate sub-results. Please refer to the following process for details:

[0075] Where c1 = a1 × b11 + a2 × b12 + an × b1n; ...; cq = a1 × bq1 + a2 × bq2 + an × bqn. [c1 ... cq] represents multiple intermediate sub-results, where cq represents the qth intermediate sub-result.

[0076] A set of image data and multiple sets of weight data are weighted to obtain multiple intermediate sub-results. One intermediate sub-result represents partial channel data of an output channel Cout, and q intermediate sub-results correspond to partial channel data corresponding to q output channels Cout.

[0077] After the weighting of a set of image data sets is completed, the multiple intermediate sub-results corresponding to the set of image data sets are stored in the data cache unit 22. Thereafter, the storage and calculation unit 23 is configured to continue to execute the process of sequentially reading the next image data set of the target image from the storage module 3, and repeat until the intermediate sub-results corresponding to all the image data sets of the target image are output, and multiple sets of channel data are obtained and stored in the data cache unit 22.

[0078] Specifically, the storage and calculation unit 23 is further configured to perform the pooling process in the convolution layer, implement downsampling with a preset convolution step size (e.g., stride=2), and perform weighted processing on the next image data group until the intermediate sub-results corresponding to all image data groups of the target image are output. A set of weight data groups is used to perform weighted processing on the entire target image, and the corresponding multiple intermediate sub-results constitute a set of channel data. The weighting processes of different sets of weight data groups correspond to different sets of channel data, that is, the number of weight data groups is consistent with the number of output channels Cout.

[0079] The above is the processing process of one convolution layer, and the output of the previous convolution layer can be used as the input of the next convolution layer. Multiple sets of channel data can be stored in the data cache unit 22, which is convenient for the storage and calculation unit 23 located in the same storage and calculation module 2 as the data cache unit 22 to call. The storage and calculation unit 23 is further configured to use the multiple sets of channel data as the target image to be detected by the next network layer 201, execute the process of sequentially reading the image data group of the target image from the data cache unit 22, and determining the multiple sets of channel data, that is, determining the output result of the next convolution layer and storing it in the data cache unit 22. After passing through each network layer 201 in sequence, the target output result of the target image is output, and the target output result is stored in the storage module 3.

[0080] The target output result here can be understood as the output result of the last network layer 201, which is, for example, a fully connected layer. The principle of full connectivity is that each node in a fully connected layer is connected to all nodes in the previous network layer 201, and there is no repeated use of eigenvalues ​​multiplied by different weights. The process of the fully connected layer can also be understood as the matrix operation process described above, and the repeated parts are not repeated here.

[0081] Here, the target output result is stored in the storage module 3 so as to further realize the prediction of the perception result by using the processing module 1.

[0082] The present disclosure utilizes a neural network model to realize perception task prediction, thereby improving perception prediction accuracy. At the same time, the neural network is complex and requires a large amount of computation, which places high demands on system performance. The present disclosure utilizes a storage-computing integrated module 2 to store multiple sets of channel data in a data cache unit 22, which is convenient for the storage-computing unit 23 located in the same storage-computing integrated module 2 as the data cache unit 22 to call. Therefore, the present disclosure utilizes a storage-computing integrated module 2 to greatly avoid the energy loss caused by the data transfer process and reduce system power consumption.

[0083] In addition, for some specific convolutional layers, the multiple sets of channel data output by them need to be normalized (for example, the normalization function (BN) algorithm 12) and / or nonlinearly processed (for example, the activation function ATC algorithm 13). At this time, the multiple sets of channel data are no longer used as the target image to be detected by the next network layer 201, but the results after normalization and / or nonlinear processing are used as the target image to be detected by the next network layer 201.

[0084] The storage and calculation unit 23 is configured to sequentially read the image data sets of the target image from the storage module 3 in response to the first instruction cached in the instruction cache unit 21. If the image data set contains a small amount of data, the storage and calculation unit 23 can directly process the image data set after reading it from the storage module 3. Of course, the read image data set can also be cached in the data cache unit 22 and then read from the data cache unit 22 according to the instruction when processing is required.

[0085] In some embodiments, as shown in Figures 2 and 3, the storage and computing unit 23 is specifically configured to respond to a first instruction cached in the instruction cache unit 21 and sequentially read the image data sets of the target image from the storage module 3. After reading a set of image data sets, the image data sets may be first stored in the data cache unit 22. Thereafter, the storage and computing unit 23 responds to a second instruction cached in the instruction cache unit 21 and reads the image data sets of the target image from the data cache unit 22. Here, the second instruction may be a pre-set instruction based on the processing capability of the storage and computing unit 23 and the data volume of the image data sets. For example, if the data volume of the image data sets is greater than the maximum data volume that the storage and computing unit 23 can retrieve, the second instruction may be an instruction to instruct the feature values ​​in the image data sets to be read in batches. If the data volume of the image data sets is less than or equal to the maximum data volume that the storage and computing unit 23 can retrieve, the second instruction may be an instruction to instruct the feature values ​​in the image data sets to be read all at once. Afterwards, the storage and computing unit 23 continues to be configured to respond to the third instruction cached in the instruction cache unit 21, and uses the first network layer 201 to perform weighted processing on the image data group according to multiple sets of pre-stored weight data groups, obtain multiple intermediate sub-results, and store them in the data cache unit 22.

[0086] Here, the second instruction can be, for example, an instruction pre-cached by the instruction cache unit 21. Exemplarily, the processing module 1 is configured to respond to the command sent by the storage and calculation unit 23 to store the image data group in the data cache unit 22, read the second instruction from the instruction cache unit 21 and parse it; the storage and calculation unit 23 responds to the second instruction parsed by the processing module 1 and executes the subsequent process. When the storage and calculation unit 23 is executed, it can instruct the processing module 1 to perform other operations by means of an interrupt. The "other operations" here can be, for example, the operation of the subsequent processing module 1 reading the third instruction, or the process of the processing module 1 performing some intermediate calculations internally and then notifying the storage and calculation unit 23 to execute subsequent instructions.

[0087] Here, the third instruction can be, for example, an instruction pre-cached by the instruction cache unit 21. Exemplarily, the processing module 1 is configured to respond to a command sent by the storage and calculation unit 23 that has read the image data group of the target image from the data cache unit 22, read the third instruction from the instruction cache unit 21 and parse it; the storage and calculation unit 23 executes subsequent processes in response to the third instruction parsed by the processing module 1. After the storage and calculation unit 23 completes execution, it can instruct the processing module 1 to perform other operations through an interrupt. The "other operations" here can be, for example, the subsequent operation of the processing module 1 reading the fourth instruction, or the process of the processing module 1 performing some intermediate calculations internally and then notifying the storage and calculation unit 23 to execute subsequent instructions. Exemplarily, taking the example of the data volume of the image data group being equal to the maximum data volume that the storage and calculation unit 23 can retrieve, a set of image data includes n eigenvalues, and a set of weight data includes n weight values; the i-th eigenvalue corresponds to the i-th weight value; n and i are both positive integers, and i is less than or equal to n. The storage and calculation unit 23 responds to the second instruction cached in the instruction cache unit 21, and reads n eigenvalues ​​in the image data group from the data cache unit 22 in sequence; responds to the third instruction cached in the instruction cache unit 21, and uses the first network layer 201 to weight the i-th eigenvalue and the i-th weight value in the weight data group for each weight data group in the multiple weight data groups to obtain an intermediate sub-result, and stores it in the data cache unit 22.

[0088] Taking the maximum amount of data that the storage and computing unit 23 can retrieve as 16 as an example, it is known that the target image is a 3D image, then the number of channels Cin = 3; the convolution kernel is 2×2; a set of image data groups includes 3×2×2 eigenvalues, such as [a11 a12 a13 a14 a15 a16 a17 a18 a19 a110 a111 a112]; the weight data group includes 3×2×2 weight values, such as [b11 b12 b13 b14 b15 b16 b17 b18 b19 b110 b111 b112]; here, reading 12 eigenvalues ​​at a time is within the range of the maximum amount of data that the storage and computing unit 23 can retrieve, so the second instruction can instruct the computing unit to read the 12 eigenvalues ​​in a set of image data groups from the data cache unit 22 in sequence. If there are 4 sets of weight data groups, they can be expressed using a matrix as follows:

[0089] When weighting the image data set using the four weight data sets above, matrix operations can be performed. For details, see the following process:

[0090] Among them, c1=a11×b11+a12×b12+a13×b13+a14×b14×a15×b15+a16×b16+a17×b17+a18×b18+a19×b19+a110×b110+a111×b111+a112×b112; c2=a11×b21+a12×b22+a13×b23+a14×b24+a15×b25+a16×b26+a17×b27+a18×b28+a19×b29+a110×b210+a111×b211+a112×b21 2; c3=a11×b31+a12×b32+a13×b33+a14×b34+a15×b35+a16×b36+a17×b37+a18×b38+a19×b39+a110×b310+a111×b311+a112×b312 ;c4=a11×b41+a12×b42+a13×b43+a14×b44+a15×b45+a16×b46+a17×b47+a18×b48+a19×b49+a110×b410+a111×b411+a112×b412.

[0091] It should be noted that the number of channels of the target image, the size of the convolution kernel, and the number of weight data groups are all for schematic illustration only and are not intended to limit the actual convolution scale.

[0092] The storage and calculation unit 23 continues to be configured to execute the process of sequentially reading the next image data group of the target image from the storage module 3, and repeat until the intermediate sub-results corresponding to all image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit 22.

[0093] Specifically, the storage and calculation unit 23 is further configured to perform the pooling process in the convolution layer, implement downsampling with a preset convolution stride, and perform weighted processing on the next image data group until the intermediate sub-results corresponding to all image data groups of the target image are output. A set of weight data groups is used to perform weighted processing on the entire target image, and the corresponding multiple intermediate sub-results constitute a set of channel data. The weighting processes of different sets of weight data groups correspond to different sets of channel data, that is, the number of weight data groups is consistent with the number of output channels Cout.

[0094] The storage and calculation unit 23 is further configured to use the multiple groups of channel data as the target image to be detected by the next network layer 201, execute the process of sequentially reading the image data group of the target image from the data cache unit 22, and determining the multiple groups of channel data until the target output result of the target image is output after passing through each network layer 201 in sequence, and the target output result is stored in the storage module 3.

[0095] In some embodiments, the amount of data in the image data set is greater than the maximum amount of data that can be retrieved by the storage and computing unit 23. The second instruction may be a pre-set instruction based on the processing capability of the storage and computing unit 23 and the amount of data in the image data set. Since the amount of data in the image data set is greater than the maximum amount of data that can be retrieved by the storage and computing unit 23, the second instruction instructs to read the feature values ​​in the image data set in batches.

[0096] Figure 5 is a schematic diagram of data interaction of the storage and computing integrated module provided by an embodiment of the present disclosure. As shown in Figure 5, the storage and computing unit 23 is specifically configured to respond to the first instruction cached in the instruction cache unit 21, read the image data group of the target image from the storage module 3 in sequence, and store it in the data cache unit 22; respond to the second instruction cached in the instruction cache unit 21, (the first batch) read the first data subset in the image data group from the data cache unit 22; the data amount of the first data subset is the maximum data amount; respond to the third instruction cached in the instruction cache unit 21, use the first network layer 201 to weight the first data subset according to the first part of the weight data of each weight data group in the multiple pre-stored weight data groups, and obtain multiple intermediate partial sub-results. (Second batch) Read a second data subset in the image data group from the data cache unit 22, where the data of the second data subset is the remaining image data in the image data group except the first data subset; use the multiple intermediate partial sub-results as initial values ​​when weighting the second data subset with the multiple sets of weight data groups, and weight the second data subset according to the multiple intermediate partial sub-results and the second part of the weight data of each weight data group in the multiple pre-stored weight data groups to obtain multiple intermediate sub-results, and store them in the data cache unit 22; continue to execute the process of sequentially reading the next image data group of the target image from the storage module 3, and repeat until the intermediate sub-results corresponding to all image data groups of the target image are output, and multiple sets of channel data are obtained and stored in the data cache unit 22; use the multiple sets of channel data as the target image to be detected by the next network layer 201, execute the process of sequentially reading the image data group of the target image from the data cache unit 22, and determine the multiple sets of channel data until the target output result of the target image is output after passing through each network layer 201 in sequence, and store the target output result in the storage module 3.

[0097] Here, the amount of data in the second data subgroup is less than or equal to the maximum amount of data that can be retrieved by the storage and calculation unit 23. The sum of the amounts of data in the first and second data subgroups is equal to the amount of data in the image data group. The amount of data in the first data subgroup can be the maximum amount of data that can be retrieved by the storage and calculation unit 23; the amount of data in the second data subgroup is the amount of data in the image data group minus the amount of data in the first data subgroup.

[0098] If the data volume of the second data subgroup is greater than the maximum data volume that the storage and computing unit 23 can retrieve, the third data subgroup will continue to be divided until the sum of the data volumes of all data subgroups is the same as the image data group, and the data volume of any data subgroup is less than or equal to the maximum data volume that the storage and computing unit 23 can retrieve.

[0099] Exemplarily, a set of image data includes n eigenvalues, and a set of weight data includes n weight values; the i-th eigenvalue corresponds to the i-th weight value; n and i are both positive integers, and i is less than or equal to n. The storage and calculation unit 23 responds to the second instruction cached in the instruction cache unit 21, and sequentially reads the n1 eigenvalues ​​in the second data subset from the data cache unit 22; responds to the third instruction cached in the instruction cache unit 21, and uses the first network layer 201 to weight the first part of the weight data of each weight data group in the multiple weight data groups, and obtains an intermediate sub-result. i1 is less than or equal to n1. The first part of the weight data of the q weight data groups includes n1×p weight values. The n1 eigenvalues ​​in the second data subgroup are read sequentially from the data cache unit 22, where n2+n1=n; the multiple intermediate sub-results are used as initial values ​​when weighting the second data subgroup with the multiple weight data groups, and for the second part of the weight data of each weight data group in the multiple weight data groups, the intermediate sub-result as the initial value, the i2th eigenvalue and the i2th weight value in the weight data group are weighted to obtain the intermediate sub-result, and stored in the data cache unit 22; n2 and i2 are both positive integers, and i2 is less than or equal to n2.

[0100] Taking the maximum amount of data that the storage and calculation unit 23 can retrieve as 16 as an example, it is known that the target image is a 3D image, then the number of channels Cin = 3; the convolution kernel is 3×3; a group of image data groups includes 3×3×3 eigenvalues, such as [a11 a12 … a127]; the weight data group includes 3×2×2 weight values, such as [b11 b12 … b127]; here, the 27 eigenvalues ​​in a group of image data groups are greater than the maximum amount of data that the storage and calculation unit 23 can retrieve. Therefore, the second instruction can instruct the calculation unit to read the eigenvalues ​​in the image data group in batches, for example, reading n1 = 16 eigenvalues ​​for the first time and n2 = 11 eigenvalues ​​for the second time. If there are q groups of weight data groups, the first part of the weight data of the q groups of weight data groups is represented by a matrix:

[0101] When weighting the first data subgroup [a11 a12 ... a116] using the first part of the weight data of the q weight data groups, a matrix operation can be performed. For details, see the following process:

[0102] Wherein, d1 = a11 × b11 + ... + a116 × b116; ...; dq = a11 × bq1 + ... + a116 × bq16. [d1 ... dq] represents multiple intermediate partial molecular results, where cq represents the qth intermediate partial molecular result.

[0103] The second part of the weight data of the q group of weight data is expressed as a matrix:

[0104] Before weighting the second data subgroup, the multiple intermediate partial sub-results are used as initial values ​​for weighting the second data subgroup using the multiple weight data groups, and the second data subgroup [a117 a118 ... a127] is weighted using the second part of the weight data of the q weight data groups. For details, see the following process:

[0105] Here, e1 = d1 + a117 × b117 + ... + a127 × b127; ...; eq = dq + a117 × bq17 + ... + a127 × bq27. [e1 ... eq] represents multiple intermediate sub-results, where eq represents the qth intermediate sub-result.

[0106] FIG6a is a schematic diagram of a storage-computation integrated circuit provided in an embodiment of the present disclosure, and FIG6b is a schematic diagram of a feature weighted shift provided in an embodiment of the present disclosure. As shown in FIG6a and FIG6b, the storage-computation integrated circuit includes a processing circuit 61 and a shift-accumulation circuit 62. The processing circuit 61 can be used to weight the first data subset according to the first part of the weight data of each weight data group in a plurality of pre-stored weight data groups, to obtain a plurality of intermediate sub-group results. Afterwards, the shift-accumulation circuit 62 (Shift&Add) is used to shift to weight the second data subset. In this process, each intermediate sub-group result is shifted as the initial value when the second data subset corresponds to each weight data group when weighting the second part of the weight data.

[0107] Here, the integrated storage and computing circuit uses an integrated storage and computing architecture to solve the storage wall problem. There is no need to specifically consider the dynamic real-time control and optimization of data flow. Its resource allocation, such as computing resources, storage resources, and input / output interface bandwidth resources, is static and can be reasonably optimized through tools. This is very beneficial for future product deployment and greatly reduces the difficulty and cost of platform migration. The calculation coefficients (such as vector or matrix coefficients, neural network coefficients) are written into the storage and computing unit 23. Taking the memristor as an example, the programmable resistor 611 is used to convert the input data into a voltage value through digital-to-analog conversion. The voltage value generates a current through the resistor, and the current is then converted from analog to digital to the calculation result. In this way, according to Ohm's law and Kirchhoff's law, the multiplication and accumulation calculation is realized, completing the analog calculation process. In addition, SRAM, FLASH, MRAM, etc. can also use similar principles for calculation. Some devices can also use digital calculation methods with digital circuits and use logic devices for storage and multiplication and addition. However, the basic principles are similar, mainly completing the multiplication and addition operations while storing.

[0108] FIG7 is a schematic diagram of convolution deployment and allocation provided by an embodiment of the present disclosure. As shown in FIG7 , a set of image data sets and multiple sets of weight data sets are weighted to obtain multiple intermediate sub-results; wherein, one intermediate sub-result represents partial channel data of one output channel Cout, and q intermediate sub-results correspond to partial channel data corresponding to q output channels Cout. The intermediate sub-results corresponding to all image data sets of the target image constitute multiple sets of channel data, and the number of weight data sets is consistent with the number of output channels Cout, that is, the number of channel data sets.

[0109] As shown in Figure 3, the output results of some convolutional layers also need to be normalized and nonlinearly processed. The normalization function algorithm 12 can be used to achieve data normalization, and the activation function algorithm 13 can be used to achieve data nonlinear processing. At this time, the multiple sets of channel data output by the convolutional layer also need to be normalized and nonlinearly processed by the processing module 1, and the results after normalization and nonlinear processing serve as the input of the next convolutional layer.

[0110] In some embodiments, the neural network model further includes a normalization function algorithm 12 and an activation function algorithm 13; the normalization function algorithm 12 and the activation function algorithm 13 are both integrated in the processing module 1.

[0111] The storage and calculation unit 23 is configured to respond to the first instruction cached in the instruction cache unit 21, read the image data group of the target image from the storage module 3 in sequence, and use the first network layer 201 to perform weighted processing on the image data group according to multiple sets of weight data groups stored in advance to obtain multiple intermediate sub-results, and store them in the data cache unit 22; continue to execute the process of sequentially reading the next image data group of the target image from the storage module 3, and repeat until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple sets of channel data are obtained and stored in the data cache unit 22.

[0112] The processing module 1 is configured to read multiple sets of channel data, perform normalization processing using the normalization function algorithm 12, and obtain a normalized result; the normalized result is processed using the activation function algorithm 13 to obtain an activation result, and is stored in the data cache unit 22; the activation result is the target image to be detected in the next network layer 201. Here, the processing process of the normalization function algorithm 12 can be, for example, the processing process corresponding to the batch normalization (BN) layer in the neural network model. The processing process of the activation function algorithm 13 can be, for example, the processing process corresponding to the linear rectification function (Relu function) in the neural network model. The Relu function is also a piecewise function. For this type of function pair, it is only necessary to set the negative value to zero when extracting the result.

[0113] The storage and calculation unit 23 is configured to respond to the fourth instruction cached in the instruction cache unit 21, and use the next network layer 201 to execute the process of sequentially reading the image data group of the target image (that is, the activation result corresponding to the previous convolution layer) from the data cache unit 22, and determining multiple groups of channel data until the target output result of the target image is output after passing through each network layer 201 in sequence, and the target output result is stored in the storage module 3. Here, the process of the storage and calculation unit 23 executing the fourth instruction to determine the output result of the next network layer 201 and the subsequent sequentially connected network layers 201 can be referred to the processing process of the first network layer 201 mentioned above. The convolution principle of each convolution layer is the same, and the repeated parts will not be repeated.

[0114] Here, the fourth instruction may be, for example, an instruction pre-cached by the instruction cache unit 21. Exemplarily, the processing module 1 is configured to respond to a command sent by the storage and calculation unit 23 indicating that the multi-channel data has been stored in the data cache unit 22, read the fourth instruction from the instruction cache unit 21, and parse it; the storage and calculation unit 23 executes subsequent processes in response to the fourth instruction parsed by the processing module 1. In some embodiments, to improve the perception and detection efficiency of a frame of an image to be detected, the image to be detected is divided into multiple sub-images. If, after traversing some of the sub-images, the perception and detection results obtained meet preset conditions, the subsequent detection tasks for the current frame of the image to be detected can be terminated, thereby improving perception and detection efficiency and reducing system power consumption. The processing module 1 is further configured to obtain the image to be detected captured by the sensing device at preset time intervals; divide the image to be detected into multiple sub-images based on the device information of the sensing device, and determine the order in which each sub-image is to be detected; and store the multiple sub-images corresponding to the image to be detected in the storage module 3. Here, the sensing device is, for example, a camera; and the image to be detected is a complete frame of image captured by the camera.

[0115] The processing module 1 can obtain the current image captured by the camera as the image to be detected at intervals of a certain period, such as 500ms (the interval can be appropriately extended to save system power). The image acquisition interval can be set according to the actual application scenario. For example, for a gaze detection task, the interval at which the processing module 1 obtains the image to be detected can be set based on the time interval between the human eye's gaze state and the non-gaze state.

[0116] The device information of the sensing device includes, for example, a variety of shooting angles of view of the shooting device and the size of the collected image. The shooting angles include, for example, a normal angle of view, a close-up (angle of view smaller than the normal shooting angle of view), a mid-range (angle of view larger than the normal angle of view), a distant view (angle of view larger than the mid-range), and the like. Different shooting angles have different corresponding image division methods. For example, the degree of refinement of the image to be detected corresponding to the distant view is greater than the degree of refinement of the image to be detected corresponding to the mid-range, the degree of refinement of the image to be detected corresponding to the mid-range is greater than the degree of refinement of the image to be detected under the normal angle of view, and the degree of refinement of the image to be detected under the normal angle of view is greater than the degree of refinement of the image to be detected corresponding to the close-up.

[0117] In one case, the image to be detected can be divided into a preset number of equal parts, with the sub-images not overlapping each other, to determine the order in which the sub-images are to be detected. Alternatively, in another case, the processing module 1 divides the image to be detected into multiple sub-images based on the device information of the sensing device. Specifically, based on the device information of the sensing device, the image to be detected is divided into the 1st to mth sub-images, with the i+1th sub-image covering the ith sub-image, and the center point of each sub-image is located in the central area of ​​the image to be detected; the order in which the 1st to mth sub-images are to be detected is 1 to m.

[0118] Figure 8 is a schematic diagram of an image block preprocessing provided by an embodiment of the present disclosure. As shown in Figure 8, taking the gaze detection task as an example, most people stare at the screen. For example, in a working state, the probability of staring at the center of the screen is greater. In order to improve the task detection efficiency, it is determined in the shortest time whether someone is staring at the screen. When dividing the image to be detected, the center area is used as the key detection position, and the image to be detected is divided into the 1st to mth sub-images, and the i+1th sub-image covers the ith sub-image. The center point of each sub-image is located in the central area of ​​the image to be detected, and the order of detection of the 1st to mth sub-images is 1 to m.

[0119] The size of the divided sub-images often does not meet the convolutional layer input requirements, and the sub-images need to be scaled. The processing module 1 is further configured to sequentially read the sub-images from the storage module 3, adjust the size of the sub-images according to a preset ratio, obtain the target image corresponding to the sub-image, and store the target image in the storage module 3; the order in which the sub-images are to be detected is the same as the order in which the target images are to be detected.

[0120] In some embodiments, the processing module 1 is further configured to, when a perception detection result corresponding to a target image does not meet a preset condition, sequentially traverse the remaining target images in the image to be detected except for the target image that has been detected in the order to be detected, until the perception detection result meets the preset condition, and send the perception detection result to the terminal; the other target images are the target images remaining in the multiple target images corresponding to the image to be detected except for the target images in the historical traversal process; determine whether all target images corresponding to the image to be detected have been traversed, and when all target images corresponding to the image to be detected have been traversed, send the perception detection result corresponding to the last traversed target image to the terminal.

[0121] Taking the gaze detection task as an example, if the perception detection result indicates that no one is gazing at the screen, the system will sequentially traverse the next target image in the order to be detected until it is determined that the perception detection result indicates that someone is gazing at the screen. The perception detection result will be sent to the terminal, causing the terminal to actively wake up from its current sleep state; if it is in the current normal display state, the terminal will remain in normal display mode. If, after traversing all target images corresponding to the target images in the order to be detected, the perception detection result still indicates that no one is gazing at the screen, the final perception detection result for the target image will be sent to the terminal, causing the terminal to remain in sleep mode. In the current normal display state, the system will further determine whether to enter sleep mode based on the current perception detection result.

[0122] The above is a complete description of a monitoring system integrated with a storage and computing subsystem 10. Although many existing mobile devices can already capture images through the front camera and perform corresponding actions after facial recognition unlocking, such as sleep, screen off, wake up, unlock, etc., it is necessary to keep the processor (processing module 1) in the device in normal working state. Although there is also a low-power mode, the computing energy consumption of our common general-purpose processors, such as CPU, GPU, and NPU, still cannot escape the energy efficiency range under Feng's law, resulting in a certain amount of energy waste. The storage and computing subsystem 10 provided in the embodiment of the present disclosure can improve the energy efficiency by 1 to 3 orders of magnitude compared with the traditional method when performing dot multiplication in matrix operations. The use of the storage and computing subsystem 10 to handle simple perception and detection tasks has a high application prospect.

[0123] In a second aspect, an embodiment of the present disclosure further provides another monitoring system. FIG9 is a schematic structural diagram of another monitoring system provided by an embodiment of the present disclosure. As shown in FIG9 , the monitoring system includes a storage and computing subsystem 10 and a terminal 30, wherein the storage and computing subsystem 10 is in communication with the terminal 30; the storage and computing subsystem 10 includes a processing module 1, a storage and computing module 2, and a storage module 3. A neural network model is integrated in the monitoring system for performing perception task detection to control the display status of the terminal 30. The neural network model includes a multi-layer sequentially connected network layer 201 and a loss function algorithm 11. The more complex processing algorithm (such as the loss function algorithm 11) in the neural network model can be integrated into the processing module 1 of the storage and computing subsystem 10, and the multi-layer sequentially connected network layer 201 in the neural network model can be integrated into the storage and computing module 2 of the storage and computing subsystem 10. The storage and computing subsystem 10 also includes a bus, and the processing module 1 can be electrically connected to the storage and computing module 2 and the storage module 3 via the bus to achieve data transmission. The storage and computing module 2 and the storage module 3 are electrically connected via an interface protocol to achieve data transmission. The storage and computing subsystem 10 and the terminal 30 are electrically connected via an interface protocol, and information is sent and received via an interrupt request (IRQ).

[0124] The storage and computing module 2 is configured to read the target image from the storage module 3 and process it using the multi-layer sequentially connected network layer 201 to obtain the target output result. Here, the target output result can be stored in the storage module 3; alternatively, the target output result can be sent directly to the processing module 1. The present embodiment uses the example of storing the target output result in the storage module 3 as an example.

[0125] It should be noted that the storage and computing integrated module 2 in the embodiment of the present disclosure is the same as the storage and computing integrated module 2 of the monitoring system provided in the first aspect, and the repeated parts will not be repeated.

[0126] The processing module 1 is configured to read the target output result and process it using the loss function algorithm 11 to obtain a perception detection result; if the perception detection result meets a preset condition, the perception detection result is sent to the terminal 30.

[0127] It should be noted that the processing module 1 in the embodiment of the present disclosure is the same as the processing module 1 of the monitoring system provided in the first aspect, and the repeated parts are not repeated here.

[0128] The terminal 30 is configured to determine the display state based on the perception detection result.

[0129] The terminal 30 is specifically configured to, if the received perception detection result indicates that someone is staring at the screen, determine its current display state; if the current display state is normal, maintain the normal display state; if the current display state is sleep, actively wake up the terminal. Furthermore, if the received perception detection result indicates that no one is staring at the screen, determine its current display state; if the current display state is normal, further determine whether to enter sleep mode based on the current perception detection result; if the current display state is sleep, actively wake up the terminal.

[0130] The embodiment of the present disclosure adopts a storage-computing integrated subsystem 10, integrating a multi-layer sequentially connected network layer 201 into the storage-computing integrated module 2. The storage-computing integrated module 2, which integrates storage and computing, supports data operations (such as matrix operations) of the network layer 201, greatly avoiding energy loss caused by the data transfer process and reducing system power consumption. Furthermore, the monitoring system provided by the present disclosure is integrated with a neural network model, which is used to predict perception tasks. Compared with traditional recognition technology, the detection accuracy of perception tasks is improved, thereby more accurately controlling the timing of the terminal 30 entering sleep or actively waking up, further reducing system power consumption.

[0131] In some embodiments, the terminal 30 also includes a display module 32 and a main control module 31; the main control module 31 is configured to reset the sleep timer when it is determined that the perception detection result meets the preset conditions, so as to receive the perception detection result sent by the storage and computing subsystem by responding to the interrupt request; and send a wake-up request to the display module 32; when it is determined that the perception detection result does not meet the preset conditions, and the current system time is greater than the preset time from the last time the display module 32 was awakened, it sends a sleep request to the display module 32 and enters the sleep state itself; the display module 32 is configured to respond to the wake-up request and determine that the display state is normal display; respond to the sleep request and determine that the display state is sleep.

[0132] Here, the preset duration can be set based on experience. It should be noted that the preset duration should be set reasonably so that the terminal 30 can enter the sleep state reasonably in actual application scenarios. If the preset duration is too short, it will cause frequent switching between sleep and wake-up states, which is not conducive to reducing power consumption. If the preset duration is too long, the power consumption of the terminal 30 cannot be reduced to the greatest extent.

[0133] Here, the display module 32 may be, for example, a display screen, and the main control module 31 may be, for example, a system on chip (SOC). The main control module 31 and the storage and computing subsystem 10 may communicate using a protocol such as UART.

[0134] In some embodiments, Figure 10 is a schematic diagram of another monitoring system provided by an embodiment of the present disclosure. As shown in Figure 10, the monitoring system also includes a sensing device 40, and the storage and computing subsystem 10 can obtain data collected by the low-power sensing device 40 through protocols such as SPI.

[0135] The monitoring system consists of an SOC display system and an intelligent perception system. As shown in Figure 9, the intelligent perception system includes a sensing device 40 and a storage and computing subsystem 10. The sensing device 40 includes, for example, visual sensors and voice sensors. For the intelligent perception system, since the main processor adopts a storage and computing integrated processing architecture, combined with a low-power camera and MIC (microphone), the overall intelligent perception system can maintain a very small power consumption range, thereby allowing the entire monitoring system to also maintain a small power consumption range.

[0136] On the third aspect, the embodiment of the present disclosure also provides a control method for a monitoring system corresponding to the monitoring system provided in the first aspect. Since the principle of solving the problem by the method in the embodiment of the present disclosure is similar to that of the monitoring system in the first aspect, the implementation of the control method of the monitoring system can refer to the implementation of the monitoring system in the first aspect above, and the repeated parts will not be repeated.

[0137] The disclosed embodiments also provide a control method for a monitoring system, wherein the monitoring system is integrated with a neural network model; the neural network model includes a multi-layer sequentially connected network layer and a loss function algorithm; the monitoring system includes at least a storage and computing subsystem, the storage and computing subsystem being communicatively connected to a terminal; the storage and computing subsystem including a processing module, a storage and computing module, and a storage module; the multi-layer sequentially connected network layer is integrated in the storage and computing module, and the loss function algorithm is integrated in the processing module;

[0138] The storage and computing module reads the target image from the storage module and processes it using multiple sequentially connected network layers to obtain a target output result. Here, the target output result can be stored in the storage module; alternatively, the target output result can be sent directly to the processing module. This embodiment of the disclosure uses the example of storing the target output result in the storage module.

[0139] The processing module reads the target output result and processes it using the loss function algorithm to obtain the perception detection result; when the perception detection result meets the preset conditions, the perception detection result is sent to the terminal so that the terminal determines the display state based on the perception detection result.

[0140] The disclosed embodiment uses a storage-computing subsystem that integrates multiple sequentially connected network layers into the storage-computing module. This system architecture, which integrates storage and computing, supports data operations (such as matrix operations) at the network layer, greatly avoiding energy loss caused by the data transfer process and reducing system power consumption. Furthermore, the control method of the monitoring system provided by the present disclosure uses a neural network model to predict perception tasks. Compared with traditional recognition technology, it improves the detection accuracy of perception tasks, thereby more accurately controlling the timing of the terminal entering sleep or actively waking up, further reducing system power consumption.

[0141] In some embodiments, for step S12, the storage and computing module includes an instruction cache unit, a data cache unit, and a storage and computing unit; the storage and computing unit is integrated with a multi-layer sequentially connected network layer.

[0142] The storage and calculation unit responds to the first instruction cached in the instruction cache unit, sequentially reads the image data group of the target image from the storage module, and uses the first network layer to perform weighted processing on the image data group according to the multiple sets of weight data groups stored in advance to obtain multiple intermediate sub-results, and stores them in the data cache unit; continues to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeats until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple sets of channel data are obtained and stored in the data cache unit; uses the multiple sets of channel data as the target image to be detected in the next network layer, executes the process of sequentially reading the image data group of the target image from the data cache unit, determines the multiple sets of channel data, until the target output result of the target image is output after passing through each network layer in sequence, and stores the target output result to the storage module.

[0143] In some embodiments, for step S12, the storage and calculation unit specifically includes: responding to the first instruction cached in the instruction cache unit, sequentially reading the image data group of the target image from the storage module, and storing it in the data cache unit; responding to the second instruction cached in the instruction cache unit, reading the image data group of the target image from the data cache unit; responding to the third instruction cached in the instruction cache unit, using the first network layer to perform weighted processing on the image data group according to multiple sets of weight data groups stored in advance, to obtain multiple intermediate sub-results, and store them in the data cache unit; continuing to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeating until the intermediate sub-results corresponding to all image data groups of the target image are output, and multiple sets of channel data are obtained and stored in the data cache unit; using the multiple sets of channel data as the target image to be detected in the next network layer, executing the process of sequentially reading the image data group of the target image from the data cache unit, determining the multiple sets of channel data, until the target output result of the target image is output after passing through each network layer in sequence, and storing the target output result in the storage module.

[0144] In some embodiments, for step S12, a group of image data includes n eigenvalues, and a group of weight data includes n weight values; the i-th eigenvalue corresponds to the i-th weight value; n and i are both positive integers, and i is less than or equal to n; the storage and calculation unit specifically includes: responding to the first instruction cached in the instruction cache unit, sequentially reading the image data group of the target image from the storage module, and storing it in the data cache unit; responding to the second instruction cached in the instruction cache unit, sequentially reading the n eigenvalues ​​in the image data group from the data cache unit; responding to the third instruction cached in the instruction cache unit, using the first network layer, for each weight data group in the multiple weight data groups, The i-th eigenvalue is weighted with the i-th weight value in the weight data group to obtain an intermediate sub-result, and the result is stored in the data cache unit; the process of sequentially reading the next image data group of the target image from the storage module is continued, and repeated until the intermediate sub-results corresponding to all image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit; the multiple groups of channel data are used as the target image to be detected in the next network layer, and the process of sequentially reading the image data group of the target image from the data cache unit is executed to determine the multiple groups of channel data until the target output result of the target image is output after passing through each network layer in sequence, and the target output result is stored in the storage module.

[0145] In some embodiments, for step S12, the amount of data in the image data group is greater than the maximum amount of data that can be retrieved by the storage and computing unit; the storage and computing unit specifically includes: responding to the first instruction cached in the instruction cache unit, sequentially reading the image data group of the target image from the storage module, and storing it in the data cache unit; responding to the second instruction cached in the instruction cache unit, reading the first data subset in the image data group from the data cache unit; the data amount of the first data subset is the maximum data amount; responding to the third instruction cached in the instruction cache unit, using the first network layer, weighting the first data subset according to the first part of the weight data of each weight data group in the multiple pre-stored weight data groups, to obtain multiple intermediate partial sub-results; reading the second data subset in the image data group from the data cache unit, the data of the second data subset is the remaining image data in the image data group except the first data subset. ; Use multiple intermediate sub-results as initial values ​​when weighting the second data sub-group with multiple groups of weight data groups, and weight the second data sub-group according to the multiple intermediate sub-results and the second part of the weight data of each weight data group in the multiple groups of weight data groups stored in advance, to obtain multiple intermediate sub-results, and store them in the data cache unit; continue to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeat until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit; use the multiple groups of channel data as the target image to be detected in the next network layer, execute the process of sequentially reading the image data group of the target image from the data cache unit, determine the multiple groups of channel data, until the target output result of the target image is output after passing through each network layer in sequence, and store the target output result in the storage module.

[0146] In some embodiments, for step S12, a group of image data includes n eigenvalues, and a group of weight data includes n weight values; the i-th eigenvalue corresponds to the i-th weight value; n and i are both positive integers, and i is less than or equal to n; the storage and calculation unit specifically includes: responding to the first instruction cached in the instruction cache unit, sequentially reading the image data group of the target image from the storage module, and storing it in the data cache unit; responding to the second instruction cached in the instruction cache unit, sequentially reading the n1 eigenvalues ​​in the first data subset from the data cache unit; responding to the third instruction cached in the instruction cache unit, using the first network layer, for the first part of the weight data of each weight data group in the multiple groups of weight data groups, weighting the i1 th eigenvalue and the i1 th weight value in the weight data group to obtain an intermediate sub-result; n1 and i1 are both positive integers, and i1 is less than or equal to n1; sequentially reading the n2 eigenvalues ​​in the second data subset from the data cache unit, n2+n1 =n; using multiple intermediate sub-results as initial values ​​when weighting the second data sub-group with multiple sets of weight data groups, for the second part of the weight data of each weight data group in the multiple sets of weight data groups, weighting the intermediate sub-results as the initial values, the i2th eigenvalue and the i2th weight value in the weight data group to obtain intermediate sub-results, and storing them in the data cache unit; n2 and i2 are both positive integers, and i2 is less than or equal to n2; continuing to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeating until the intermediate sub-results corresponding to all image data groups of the target image are output, and multiple sets of channel data are obtained and stored in the data cache unit; using the multiple sets of channel data as the target image to be detected in the next network layer, executing the process of sequentially reading the image data group of the target image from the data cache unit, determining the multiple sets of channel data, until the target output result of the target image is output after passing through each network layer in sequence, and storing the target output result in the storage module.

[0147] In some embodiments, the neural network model further includes a normalization function algorithm and an activation function algorithm; the normalization function algorithm and the activation function algorithm are both integrated in the processing module.

[0148] The control method of the monitoring system specifically includes: the storage and calculation unit responds to the first instruction cached in the instruction cache unit, sequentially reads the image data group of the target image from the storage module, uses the first network layer to perform weighted processing on the image data group according to multiple pre-stored weight data groups, obtains multiple intermediate sub-results, and stores them in the data cache unit; continues to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeats until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit; the processing module reads the multiple groups of channel data, performs normalization processing using a normalization function algorithm, obtains a normalized result; processes the normalized result using an activation function algorithm, obtains an activation result, and stores it in the data cache unit; the activation result is the target image to be detected in the next network layer; the storage and calculation unit responds to the fourth instruction cached in the instruction cache unit, uses the next network layer to execute the process of sequentially reading the image data group of the target image from the data cache unit, determines multiple groups of channel data, and outputs a target output result of the target image after sequentially passing through each network layer, and stores the target output result in the storage module.

[0149] The processing module is further configured to obtain an image to be detected captured by a sensing device at a preset time interval; divide the image to be detected into multiple sub-images based on device information of the sensing device, and determine the order in which each sub-image is to be detected; store the multiple sub-images corresponding to the image to be detected in a storage module; and read the sub-images from the storage module in sequence, adjust the size of the sub-images according to a preset ratio, obtain a target image corresponding to the sub-image, and store the target image in the storage module; the order in which each sub-image is to be detected is the same as the order in which each target image is to be detected.

[0150] In some embodiments, the processing module divides the image to be detected into multiple sub-images according to the device information of the sensing device. Specifically, according to the device information of the sensing device, the image to be detected is divided into the 1st to mth sub-images, and the i+1th sub-image covers the i-th sub-image, and the center point of each sub-image is located in the central area of ​​the image to be detected; the order to be detected from the 1st to the mth sub-image is 1 to m.

[0151] In some embodiments, the processing module also includes: when the perception detection result corresponding to a target image does not meet the preset conditions, the remaining target images in the image to be detected except the target image that has been detected are traversed in sequence according to the order to be detected, until the perception detection result meets the preset conditions, and the perception detection result is sent to the terminal; the other target images are the target images remaining in the multiple target images corresponding to the image to be detected except the target images in the historical traversal process; it is determined whether all target images corresponding to the image to be detected have been traversed, and when all target images corresponding to the image to be detected have been traversed, the perception detection result corresponding to the last traversed target image is sent to the terminal.

[0152] On the fourth aspect, the embodiments of the present disclosure also provide a control method for a monitoring system corresponding to the monitoring system provided in the second aspect. Since the principle of solving the problem by the method in the embodiments of the present disclosure is similar to that of the monitoring system in the second aspect, the implementation of the control method for the monitoring system can refer to the implementation of the monitoring system in the second aspect above, and the repeated parts will not be repeated.

[0153] The disclosed embodiments further provide another control method for a monitoring system, wherein the monitoring system is integrated with a neural network model; the neural network model includes a multi-layer sequentially connected network layer and a loss function algorithm; the monitoring system includes a storage and computing subsystem and a terminal; the storage and computing subsystem includes a processing module, a storage and computing module, and a storage module; the multi-layer sequentially connected network layer is integrated in the storage and computing module, and the loss function algorithm is integrated in the processing module;

[0154] The storage and computing module reads the target image from the storage module, processes it using multiple sequentially connected network layers to obtain the target output result, and stores the target output result in the storage module;

[0155] The processing module reads the target output result and processes it using the loss function algorithm to obtain the perception detection result; if the perception detection result meets the preset conditions, the perception detection result is sent to the terminal;

[0156] The terminal determines the display status based on the perception detection result.

[0157] The disclosed embodiment uses a storage-computing subsystem that integrates multiple sequentially connected network layers into the storage-computing module. This system architecture, which integrates storage and computing, supports data operations (such as matrix operations) at the network layer, greatly avoiding energy loss caused by the data transfer process and reducing system power consumption. Furthermore, the control method of the monitoring system provided by the present disclosure uses a neural network model to predict perception tasks. Compared with traditional recognition technology, it improves the detection accuracy of perception tasks, thereby more accurately controlling the timing of the terminal entering sleep or actively waking up, further reducing system power consumption.

[0158] In some embodiments, the terminal also includes a display module and a main control module; when the main control module determines that the perception detection result meets the preset conditions, the main control module resets the sleep timer to receive the perception detection result sent by the storage and computing subsystem by responding to the interrupt request; and sends a wake-up request to the display module; when it is determined that the perception detection result does not meet the preset conditions, and the current system time is greater than the preset time from the last time the display module was awakened, the main control module sends a sleep request to the display module and enters the sleep state itself; the display module responds to the wake-up request and determines that the display state is normal display; responds to the sleep request and determines that the display state is sleep.

[0159] Taking the gaze detection task as an example, FIG11a is an exemplary terminal workflow diagram provided by an embodiment of the present disclosure. As shown in FIG11a, the main control module: S11, starts system initialization; S12, sends a startup instruction to the storage and computing subsystem, and receives a response confirmation from the storage and computing subsystem, and executes S13 in sequence; S13, resets the sleep timer; S14, polls the serial port to see if a perception detection result indicating gaze is received, and sends a response confirmation to the storage and computing subsystem; if so, returns to step S13; if No, execute S15; S15, determine whether the preset time is exceeded; if so, execute S16; if not, return to execute S14; S16, the main control SOC enters the sleep state, and sends a sleep request to the display module, and executes S17 in sequence; S17, polls the serial port to see whether it receives the perception detection result indicating gaze, and sends a response confirmation to the storage and computing subsystem; if so, execute S18; if not, repeat S17; S18, wake up the main control SOC, send a wake-up request to the display module, and return to step S13.

[0160] Figure 11b is an exemplary workflow diagram of a storage and computing subsystem provided in an embodiment of the present disclosure. As shown in Figure 11b, the storage and computing subsystem: S101, starts system initialization; S102, polls the serial port to identify the gaze detection instruction sent by the main control module, and sends a response confirmation to the main control module, and executes S103 in sequence; S103, the processing module obtains the image to be detected captured by the shooting device at a preset time interval; S104, the processing module divides the image to be detected into multiple sub-images; S105, the processing module stores the multiple sub-images into the storage module; S106, the processing module extracts the sub-images from the storage module for pre-processing to obtain the target image, for example, including RGB signal conversion and image scaling; S107, the processing module stores the target image into the storage module; S108, the storage and computing module responds to the first instruction cached in the instruction cache unit, reads the image data group of the target image from the storage module, and stores it in the data cache unit. It should be noted that if the current execution subject has other intermediate convolution layers except the first convolution layer, the storage and computing module reads the image data group of the target image from the data cache unit. S109, the neural network model performs perception prediction (for detailed process, please refer to the calculation process of each network layer and loss function algorithm mentioned above, which will not be repeated here); S110, the storage and computing module determines whether the output is the target output result or the intermediate sub-result. If the output is the intermediate sub-result, execute S111. If the output is the target output result, execute S112; S111, store the intermediate sub-result in the data cache unit, and return to execute S108; S112, store the target output result in the storage module; S113, the processing module reads the target output result from the storage module, and uses Use the loss function algorithm to calculate the loss and obtain the perception detection result; S114, determine whether the perception detection result indicates that someone is staring at the screen, if not, execute S115, if yes, execute S116; S115, determine whether the target image corresponding to the image to be detected has been traversed, if yes, execute S116, if no, return to execute S103; S116, send the perception detection result to the terminal, and receive a response confirmation; S117, determine whether the stop perception detection task sent by the terminal is received, if yes, return to execute S102, if not, return to execute S103.

[0161] In a fifth aspect, according to an embodiment of the present disclosure, a non-transitory computer-readable storage medium is further provided. The non-transitory computer-readable storage medium stores a computer program, wherein when the program is executed by a processor, the program implements the steps of the control method of the monitoring system as described in any of the above embodiments.

[0162] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the system of the present disclosure are executed.

[0163] It should be noted that the computer non-transitory readable medium shown in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any non-transitory computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the non-transitory computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the aforementioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two connected boxes can actually represent execution in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] It is understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Those skilled in the art may make various modifications and improvements without departing from the spirit and substance of the present disclosure, and such modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A monitoring system, wherein a neural network model is integrated; the neural network model includes a multi-layer sequentially connected network layer and a loss function algorithm; the monitoring system at least includes a storage-computing integrated subsystem, and the storage-computing integrated subsystem is communicatively connected to a terminal; the storage-computing integrated subsystem includes a processing module, a storage-computing integrated module and a storage module; the multi-layer sequentially connected network layer is integrated in the storage-computing integrated module, and the loss function algorithm is integrated in the processing module; The storage-computation integrated module is configured to read the target image from the storage module and process it using multiple sequentially connected network layers to obtain a target output result; The processing module is configured to read the target output result and process it using a loss function algorithm to obtain a perception detection result; In a case where the perception detection result meets a preset condition, the perception detection result is sent to the terminal, so that the terminal determines a display state based on the perception detection result.

2. The monitoring system according to claim 1, wherein: The storage-computing integrated module includes an instruction cache unit, a data cache unit and a storage-computing unit; the storage-computing unit is integrated with a multi-layer sequentially connected network layer; The storage and calculation unit is configured to respond to the first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, use the first layer of the network layer to perform weighted processing on the image data group according to multiple groups of weight data groups stored in advance, obtain multiple intermediate sub-results, and store them in the data cache unit; continue to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeat until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit; use the multiple groups of channel data as the target image to be detected in the next layer of the network layer, execute the process of sequentially reading the image data group of the target image from the data cache unit, and determine the multiple groups of channel data until the target output result of the target image is output after passing through each of the network layers in sequence.

3. The monitoring system according to claim 2, wherein: The storage and calculation unit is specifically configured to respond to the first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, and store them in the data cache unit; In response to the second instruction cached in the instruction cache unit, the image data group of the target image is read from the data cache unit; in response to the third instruction cached in the instruction cache unit, the image data group is weighted according to the plurality of weight data groups stored in advance using the first layer of the network layer, to obtain a plurality of intermediate sub-results, and the intermediate sub-results are stored in the data cache unit; the process of sequentially reading the next image data group of the target image from the storage module is continued, and repeated until the intermediate sub-results corresponding to all the image data groups of the target image are output, and a plurality of channel data groups are obtained and stored in the data cache unit; the plurality of channel data groups are used as the target image to be detected in the next layer of the network layer, and the process of sequentially reading the image data group of the target image from the data cache unit is executed to determine the plurality of channel data groups, until the target output result of the target image is output after sequentially passing through each of the network layers.

4. The monitoring system according to claim 3, wherein: A set of the image data includes n eigenvalues, and a set of the weight data includes n weight values; the i-th eigenvalue corresponds to the i-th weight value; n and i are both positive integers, and i is less than or equal to n; The storage and calculation unit is specifically configured to respond to the first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, and store them in the data cache unit; In response to a second instruction cached in the instruction cache unit, sequentially read n characteristic values ​​in the image data group from the data cache unit; In response to the third instruction cached in the instruction cache unit, using the first layer of the network layer, for each of the multiple weight data groups, weight the i-th feature value and the i-th weight value in the weight data group to obtain the intermediate sub-result, and store it in the data cache unit; continue to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeat until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit; use the multiple groups of channel data as the target image to be detected in the next layer of the network layer, execute the process of sequentially reading the image data group of the target image from the data cache unit, and determine the multiple groups of channel data until the target output result of the target image is output after passing through each of the network layers in sequence.

5. The monitoring system according to claim 3, wherein: The amount of data in the image data set is greater than the maximum amount of data that can be retrieved by the storage and calculation unit; The storage and calculation unit is specifically configured to respond to the first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, and store them in the data cache unit; In response to a second instruction cached in the instruction cache unit, reading a first data subset in the image data group from the data cache unit; The data amount of the first data subgroup is the maximum data amount; in response to the third instruction cached in the instruction cache unit, the first network layer is used to weight the first data subgroup according to the first part of the weight data of each of the plurality of weight data groups stored in advance, to obtain a plurality of intermediate sub-results; A second data subset in the image data group is read from the data cache unit, wherein the data of the second data subset is the remaining image data in the image data group except the first data subset; a plurality of the intermediate partial sub-results are respectively used as initial values ​​when the second data subset is weighted by the plurality of weight data groups, and the second data subset is weighted according to the plurality of the intermediate partial sub-results and the second part of the weight data of each of the plurality of weight data groups stored in advance, to obtain a plurality of intermediate sub-results, and store them in the data cache unit; the process of sequentially reading the next image data group of the target image from the storage module is continued, and repeated until the intermediate sub-results corresponding to all the image data groups of the target image are output, and a plurality of groups of channel data are obtained and stored in the data cache unit; the plurality of groups of channel data are used as the target image to be detected in the next layer of the network layer, and the process of sequentially reading the image data group of the target image from the data cache unit and determining the plurality of groups of channel data is performed until the target output result of the target image is output after sequentially passing through each of the network layers.

6. The monitoring system according to claim 5, wherein: A set of the image data includes n eigenvalues, and a set of the weight data includes n weight values; the i-th eigenvalue corresponds to the i-th weight value; n and i are both positive integers, and i is less than or equal to n; The storage and calculation unit is specifically configured to respond to the first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, and store them in the data cache unit; In response to a second instruction cached in the instruction cache unit, sequentially read n1 characteristic values ​​in the first data subset from the data cache unit; In response to the third instruction cached in the instruction cache unit, using the first layer of the network layer, for the plurality of groups of weight data sets The first part of the weight data of each weight data group in the weight data group, weighting the i1th eigenvalue and the i1th weight value in the weight data group to obtain the intermediate sub-result; n1 and i1 are both positive integers, and i1 is less than or equal to n1; read the n2 eigenvalues ​​in the second data sub-group from the data cache unit in sequence, n2+n1=n; use the multiple intermediate sub-results as the initial values ​​when the multiple weight data groups are weighted for the second data sub-group, and for the second part of the weight data of each weight data group in the multiple weight data groups, use the intermediate sub-result as the initial value, the i2th eigenvalue and the i2th weight value in the weight data group Weighted, to obtain the intermediate sub-result, and store it in the data cache unit; n2 and i2 are both positive integers, and i2 is less than or equal to n2; continue to execute the process of sequentially reading the next image data group of the target image from the storage module, and repeat until the intermediate sub-results corresponding to all image data groups of the target image are output, and multiple groups of channel data are obtained and stored in the data cache unit; use the multiple groups of channel data as the target image to be detected in the next layer of the network layer, execute the process of sequentially reading the image data group of the target image from the data cache unit, and determine the multiple groups of channel data, until the target output result of the target image is output after sequentially passing through each of the network layers.

7. The monitoring system according to claim 2, wherein: The neural network model also includes a normalization function algorithm and an activation function algorithm; the normalization function algorithm and the activation function algorithm are both integrated in the processing module; The storage and calculation unit is configured to respond to the first instruction cached in the instruction cache unit, sequentially read the image data group of the target image from the storage module, use the first layer of the network layer to perform weighted processing on the image data group according to the multiple sets of weight data groups stored in advance, obtain multiple intermediate sub-results, and store them in the data cache unit; continue to perform the process of sequentially reading the next image data group of the target image from the storage module, and repeat until the intermediate sub-results corresponding to all the image data groups of the target image are output, and multiple sets of channel data are stored in the data cache unit; The processing module is configured to read multiple groups of channel data, perform normalization processing using the normalization function algorithm, and obtain a normalized result; Processing the normalized result using the activation function algorithm to obtain an activation result, and storing the result in the data cache unit; The activation result is a target image to be detected by the next network layer; The storage and computing unit is configured to respond to the fourth instruction cached in the instruction cache unit, and use the next layer of the network layer to execute the process of sequentially reading the image data group of the target image from the data cache unit, determining multiple groups of channel data, until the target output result of the target image is output after passing through each of the network layers in sequence.

8. The monitoring system according to claim 1, wherein: The processing module is further configured to obtain the image to be detected collected by the sensing device at a preset time interval; divide the image to be detected into a plurality of sub-images according to the device information of the sensing device, and determine the order to be detected of each sub-image; The plurality of sub-images corresponding to the image to be detected are stored in the storage module; and the sub-images are sequentially read from the storage module, the sizes of the sub-images are adjusted according to a preset ratio, a target image corresponding to the sub-images is obtained, and the target image is stored in the storage module; the order in which the sub-images are to be detected is the same as the order in which the target images are to be detected.

9. The monitoring system according to claim 8, wherein: The processing module divides the image to be detected into multiple sub-images according to the device information of the sensing device. Specifically, according to the device information of the sensing device, the image to be detected is divided into the 1st to mth sub-images, and the i+1th sub-image covers the ith sub-image, and the center point of each sub-image is located in the central area of ​​the image to be detected; the order of detection of the 1st to mth sub-images is 1 to m.

10. The monitoring system according to claim 8 or 9, wherein: The processing module is also configured to, if the perception detection result corresponding to the target image does not meet the preset conditions, sequentially traverse the other target images remaining in the images to be detected except the target image that has been detected in accordance with the order to be detected, until the perception detection result meets the preset conditions, send the perception detection result to the terminal; the other target images are the target images remaining in the multiple target images corresponding to the image to be detected except the target images in the historical traversal process; determine whether all the target images corresponding to the image to be detected have been traversed, and if all the target images corresponding to the image to be detected have been traversed, send the perception detection result corresponding to the last traversed target image to the terminal.

11. A monitoring system, wherein a neural network model is integrated; the neural network model includes a multi-layer sequentially connected network layer and a loss function algorithm; the monitoring system includes a storage-computing integrated subsystem and a terminal, the storage-computing integrated subsystem is communicatively connected to the terminal; the storage-computing integrated subsystem includes a processing module, a storage-computing integrated module and a storage module; the multi-layer sequentially connected network layer is integrated in the storage-computing integrated module, and the loss function algorithm is integrated in the processing module; The storage-computation integrated module is configured to read the target image from the storage module and process it using multiple sequentially connected network layers to obtain a target output result; The processing module is configured to read the target output result and process it using a loss function algorithm to obtain a perception detection result; if the perception detection result meets a preset condition, send the perception detection result to the terminal; The terminal is configured to determine a display state based on the perception detection result.

12. The monitoring system according to claim 11, wherein: The terminal also includes a display module and a main control module; The main control module is configured to reset the sleep timer when it is determined that the perception detection result meets the preset condition, so as to receive the perception detection result sent by the storage and computing subsystem by responding to the interrupt request; and send a wake-up request to the display module; when it is determined that the perception detection result does not meet the preset condition, and the time from the current system time to the last time the display module was awakened is greater than the preset time, send a sleep request to the display module and enter a sleep state itself; The display module is configured to respond to the wake-up request and determine that the display state is normal display; and respond to the sleep request and determine that the display state is sleep.

13. A control method for a monitoring system, wherein: The monitoring system is integrated with a neural network model; the neural network model includes a multi-layer sequentially connected network layer and a loss function algorithm; the monitoring system at least includes a storage-computing integrated subsystem, and the storage-computing integrated subsystem is connected to the terminal for communication; the storage-computing integrated subsystem includes a processing module, a storage-computing integrated module and a storage module; the multi-layer sequentially connected network layer is integrated in the storage-computing integrated module, and the loss function algorithm is integrated in the processing module; The storage-computation integrated module reads the target image from the storage module and processes it using multiple sequentially connected network layers to obtain a target output result; The processing module reads the target output result and processes it using a loss function algorithm. Obtaining a perception detection result; and when the perception detection result meets a preset condition, sending the perception detection result to the terminal, so that the terminal determines a display state based on the perception detection result.

14. A control method for a monitoring system, wherein: The monitoring system is integrated with a neural network model; the neural network model includes a multi-layer sequentially connected network layer and a loss function algorithm; the monitoring system includes a storage-computing integrated subsystem and a terminal, and the storage-computing integrated subsystem is communicatively connected with the terminal; the storage-computing integrated subsystem includes a processing module, a storage-computing integrated module and a storage module; the multi-layer sequentially connected network layer is integrated in the storage-computing integrated module, and the loss function algorithm is integrated in the processing module; The storage-computation integrated module reads the target image from the storage module and processes it using multiple sequentially connected network layers to obtain a target output result; The processing module reads the target output result and processes it using a loss function algorithm to obtain a perception detection result; if the perception detection result meets a preset condition, the perception detection result is sent to the terminal; The terminal determines a display state based on the perception detection result.

15. A computer non-transitory readable storage medium, wherein: The computer non-transitory readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the control method of the monitoring system as claimed in claim 13 and / or 14 are executed.

Images