Control method and apparatus for broadcast monitoring system and compute-in-memory system

Abstract

The present disclosure relates to the technical fields of image recognition and terminal broadcast monitoring, and provides a control method and apparatus for a broadcast monitoring system, a computer device and a storage medium. The control method for the broadcast monitoring system comprises: acquiring an image to be detected; dividing said image into a plurality of target sub-images according to a view angle of a photographing device that photographs said image; for each target sub-image among the plurality of target sub-images, processing the target sub-image by using a pre-trained target neural network model, to obtain a recognition result of the target sub-image; and on the basis of the recognition result corresponding to each target sub-image, obtaining a detection result of whether a target object is present in said image or not, and sending the detection result to a terminal, so that the terminal determines a display state at least on the basis of the detection result.

Description

Control method, device and storage and computing integrated system for monitoring and broadcasting system Technical Field

[0001] The present disclosure belongs to the technical field of image recognition and terminal monitoring, and particularly relates to a control method and device for a monitoring system, and a storage and computing integrated system. Background Art

[0002] With the rapid development of information technology, modern electronic devices are also rapidly developing towards intelligence, lightweightness, and portability. For smart terminals, displaying surveillance images for extended periods of time results in high system power consumption, impacting system lifespan. Therefore, achieving low-power operation is a pressing technical challenge in the surveillance and broadcasting field.

[0003] Summary of the Invention

[0004] The present disclosure aims to solve at least one of the technical problems existing in the prior art and provides a control method, device and storage-computing integrated system for a monitoring and broadcasting system.

[0005] In a first aspect, the technical solution adopted to solve the technical problem of the present disclosure is a control method of a monitoring system, comprising:

[0006] Obtain the image to be detected;

[0007] Dividing the image to be detected into a plurality of target sub-images according to a shooting angle of a shooting device that shoots the image to be detected;

[0008] For each of the plurality of target sub-images, using a pre-trained target neural network model, processing the target sub-image to obtain a recognition result of the target sub-image;

[0009] Based on the recognition results corresponding to the target sub-images, a detection result of whether a target object exists in the image to be detected is obtained, and the detection result is sent to the terminal, so that the terminal determines a display state based on at least the detection result.

[0010] In some embodiments, dividing the image to be detected into a plurality of target sub-images according to the shooting angle of the shooting device includes:

[0011] When the shooting angle of view is within a preset angle of view range, the image to be detected is divided into a first sub-region, a second sub-region, and a third sub-region sequentially arranged along a first direction, and the first sub-region and the third sub-region partially overlap with the second sub-region; wherein the width of the first sub-region in the first direction is equal to the width of the third sub-region in the first direction, and the width of the second sub-region in the first direction is greater than the width of the first sub-region in the first direction; and the first direction is the height direction of the image to be detected;

[0012] Dividing the portion of the image to be detected located in the first sub-area into a plurality of first sub-images arranged side by side along the second direction, wherein at least some of the first sub-images have equal widths in the second direction; the second direction being the width direction of the image to be detected;

[0013] Dividing the portion of the image to be detected located in the second sub-area into a plurality of second sub-images arranged side by side along the second direction, wherein at least some of the second sub-images have equal widths in the second direction;

[0014] Dividing the portion of the image to be detected located in the third sub-area into a plurality of third sub-images arranged side by side along the second direction, wherein at least some of the third sub-images have equal widths in the second direction; the target sub-image includes a first sub-image, a second sub-image, and a third sub-image;

[0015] The width of the third sub-image in the second direction is greater than the width of the second sub-image in the second direction, and the width of the second sub-image in the second direction is greater than the width of the first sub-image in the second direction.

[0016] In some embodiments, the first sub-images at least partially overlap in the second direction; the second sub-images at least partially overlap in the second direction; and the third sub-images have no overlap in the second direction.

[0017] In some embodiments, the ratio of the width of the second sub-region to the width of the first sub-region in the first direction is 2:1;

[0018] The ratio of the width of the third sub-image to the width of the second sub-image in the second direction is 3:2, and the ratio of the width of the second sub-image to the width of the first sub-image in the second direction is 4:3.

[0019] In some embodiments, among the plurality of first sub-images arranged side by side along the second direction, except for the first and last first sub-images arranged side by side along the second direction, the widths of the remaining first sub-images in the second direction are the same; the widths of the first and last first sub-images in the second direction are the same and smaller than the widths of the remaining first sub-images in the second direction; and the ratio of the overlapping width of two adjacent first sub-images in the second direction to the widths of the remaining first sub-images in the second direction is 1:10;

[0020] Among the plurality of second sub-images arranged side by side along the second direction, except for the first and last second sub-images arranged side by side along the second direction, the remaining second sub-images have the same width in the second direction; the first and last second sub-images have the same width in the second direction and are smaller than the widths of the remaining second sub-images in the second direction; and the ratio of an overlapping width of two adjacent second sub-images in the second direction to the widths of the remaining second sub-images in the second direction is 1:10;

[0021] Among the plurality of third sub-images arranged side by side along the second direction, except for the first and last third sub-images arranged side by side along the second direction, the remaining third sub-images have the same width in the second direction; the first and last third sub-images have the same width in the second direction and are smaller than the widths of the remaining third sub-images in the second direction; and the ratio of the overlapping width of two adjacent third sub-images in the second direction to the widths of the remaining third sub-images in the second direction is 1:10;

[0022] The first sub-image and the second sub-image adjacent in the first direction have an overlapping width in the first direction and a width of the second sub-image in the first direction of 1:10; the third sub-image and the second sub-image adjacent in the first direction have an overlapping width in the first direction and a width of the second sub-image in the first direction of 1:10.

[0023] In some embodiments, the target neural network model is trained by the following steps:

[0024] Obtaining a first training data set and a second training data set; wherein the second training data set is obtained based on the first training data set after screening;

[0025] Training the teacher machine learning model to be trained according to the first training data set to obtain a preliminarily trained teacher machine learning model;

[0026] Training the preliminarily trained teacher machine learning model according to the second training data set to obtain a trained teacher machine learning model;

[0027] According to the second training data set and the trained teacher machine learning model, the student machine learning model to be trained is trained using a knowledge distillation training method to obtain the trained student machine learning model as the target neural network model.

[0028] In some embodiments, the second training dataset includes a plurality of training images annotated with sample labels;

[0029] The method of training the student machine learning model to be trained using a knowledge distillation training method based on the second training data set and the trained teacher machine learning model to obtain the trained student machine learning model as the target neural network model includes:

[0030] Inputting the training image into the trained teacher machine learning model to obtain a first output result of the trained teacher machine learning model;

[0031] Inputting the training image into the student machine learning model to be trained to obtain a second output result of the student machine learning model to be trained;

[0032] Determining a first loss function according to the first output result and the second output result;

[0033] Determining a second loss function according to the second output result and the sample label of the training image;

[0034] Obtaining a weighted loss function according to the first loss function and the second loss function;

[0035] The parameters of the student machine learning model to be trained are adjusted according to the weighted loss function until the weighted loss function converges, thereby obtaining the trained student machine learning model as the target neural network model.

[0036] In some embodiments, the first training data set is determined by the following steps:

[0037] Acquire an original data set; the original data set includes a plurality of initial sample images;

[0038] Identifying the target object on the initial sample image, and if the target object exists, determining a first reference frame containing the target object;

[0039] updating the position of the first reference frame according to the position information of the first reference frame to obtain a second reference frame;

[0040] determining an overlap degree between the second reference frame and the first reference frame based on the position information of the second reference frame and the position information of the first reference frame;

[0041] According to the comparison result between the overlap degree and the first preset threshold, the sample label is marked for the portion of the sample image of the initial sample image located in the second reference frame, and the portion of the sample image marked with the sample label is used as the training image in the first training data set.

[0042] In some embodiments, updating the position of the first reference frame according to the position information of the first reference frame to obtain the second reference frame includes:

[0043] moving a specific coordinate point in the first reference frame according to the position information of the first reference frame to obtain a third reference frame;

[0044] According to the position information of the third reference frame, with the center point of the third reference frame as the center, the width and height of the third reference frame are adjusted according to a preset scaling factor to obtain a fourth reference frame;

[0045] According to the position information of the fourth reference frame, a new center point is determined according to the first preset center point range; and according to the new center point and the preset cropping range, the width and height of the fourth reference frame are adjusted to obtain the second reference frame.

[0046] In some embodiments, marking the sample label for the portion of the initial sample image located in the second reference frame according to the comparison result between the overlap degree and a first preset threshold includes:

[0047] When the overlap is greater than or equal to the first preset threshold, a first preset range is generated according to the first preset probability, a second preset range is generated according to the second preset probability, a third preset range is generated according to the third preset probability, and a fourth preset range is generated according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; and the sum of the first preset probability, the second preset probability, the third preset probability, and the fourth preset probability is 1;

[0048] When the overlap degree is within the first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with the overlap degree within the first preset range and the total number of training images with the positive sample labels in the first training data set is the first preset probability;

[0049] When the overlap degree is within the second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with the overlap degree within the second preset range and the total number of training images with the positive sample labels in the first training data set is the second preset probability;

[0050] When the overlap degree is within the third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with the overlap degree within the third preset range and the total number of training images with the positive sample labels in the first training data set is equal to the third preset probability;

[0051] When the overlap degree is within the fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with the overlap degree within the fourth preset range and the total number of training images with the positive sample labels in the first training data set is equal to the fourth preset probability;

[0052] The overlap within the first preset range is smaller than the overlap within the second preset range, the overlap within the second preset range is smaller than the overlap within the third preset range, and the overlap within the third preset range is smaller than the overlap within the fourth preset range.

[0053] In some embodiments, in the step of obtaining the first training data set, if the target object is identified on the initial sample image and it is determined that the target object does not exist, the step further includes:

[0054] The initial sample image is marked with a negative sample label, and the initial sample image marked with the negative sample label is used as a training image in the first training data set.

[0055] In some embodiments, after identifying the target object on the initial sample image to determine the presence of the target object and determining a first reference frame containing the target object, the method further includes:

[0056] Determining a target center point in the initial sample image according to the size of the initial sample image and in accordance with a second preset center point range, and determining a fifth reference frame with the target center point as the center;

[0057] determining, based on the position information of the fifth reference frame and the position information of the first reference frame, a degree of overlap between the fifth reference frame and any first reference frame in the initial sample image;

[0058] If the degree of overlap between the fifth reference frame and any first reference frame in the initial sample image is less than or equal to a second preset threshold, the portion of the sample image of the initial sample image located in the fifth reference frame is marked as a negative sample label, and the portion of the sample image marked with the negative sample label is used as the training image.

[0059] In some embodiments, acquiring the image to be detected includes:

[0060] Get multiple continuous video frames captured by the camera;

[0061] Based on the multiple continuous video frames, a frame difference method is used to determine whether there is a moving target at the shooting scene of the shooting device. If there is a moving target, the video frame shot by the shooting device is used as the image to be detected.

[0062] In a second aspect, an embodiment of the present disclosure further provides a control method for a monitoring system, wherein the monitoring system includes a storage and computing integrated system and a terminal; the storage and computing integrated system includes a processing module, a storage and computing integrated module, and an external storage module; the control method for the monitoring system includes:

[0063] The processing module obtains an image to be detected; divides the image to be detected into a plurality of target sub-images according to the shooting angle of the shooting device that shoots the image to be detected, and stores the sub-images in the external storage module;

[0064] The storage and computing module reads the target sub-image; processes the target sub-image using a pre-trained target neural network model to obtain a recognition result of the target sub-image, and stores the recognition result in the external storage module;

[0065] The processing module obtains a detection result of whether a target object exists in the image to be detected based on the recognition results corresponding to each target sub-image, and sends the detection result to the terminal;

[0066] The terminal determines a display state based on the detection result.

[0067] In some embodiments, the terminal includes a display module and a main control module;

[0068] The terminal determines a display state based on the detection result, including:

[0069] The terminal determines a display state based on the detection result, including:

[0070] When the main control module determines that the detection result indicates that the target object exists in the image to be detected, the main control module sends a wake-up request to the display module and resets the sleep timer in response to the terminal request to receive the detection result sent by the processing module; when the main control module determines that the detection result indicates that the target object does not exist in the image to be detected and the time between the current system time and the last time the display module was awakened is greater than a preset time length, the main control module sends a sleep request to the display module and controls itself to enter a sleep state;

[0071] The display module responds to the wake-up request and determines that the display state is to normally display the captured image of the capture device; and responds to the sleep request and determines that the display state is to sleep.

[0072] In a third aspect, the embodiments of the present disclosure further provide a storage and computing integrated system, comprising a processing module, a storage and computing integrated module, and an external storage module;

[0073] The processing module obtains an image to be detected; divides the image to be detected into a plurality of target sub-images according to the shooting angle of the shooting device that shoots the image to be detected, and stores the sub-images in the external storage module;

[0074] The storage and computing module reads the target sub-image; processes the target sub-image using a pre-trained target neural network model to obtain a recognition result of the target sub-image, and stores the recognition result in the external storage module;

[0075] The processing module obtains a detection result of whether a target object exists in the image to be detected based on the recognition results corresponding to each target sub-image, and sends the detection result to the terminal, so that the terminal determines a display state based on the detection result.

[0076] In a fourth aspect, an embodiment of the present disclosure further provides a control device for a monitoring and broadcasting system, which includes a storage and computing integrated system and a terminal; the storage and computing integrated system includes a processing module, a storage and computing integrated module, and an external storage module;

[0077] The processing module is configured to obtain an image to be detected; divide the image to be detected into a plurality of target sub-images according to the shooting angle of the shooting device that shoots the image to be detected, and store the sub-images in the external storage module;

[0078] The storage and computing module is configured to read the target sub-image; process the target sub-image using a pre-trained target neural network model to obtain a recognition result of the target sub-image; and store the recognition result in the external storage module;

[0079] The processing module is configured to obtain a detection result of whether a target object exists in the image to be detected based on the recognition results corresponding to each target sub-image, and send the detection result to the terminal;

[0080] The terminal is configured to determine a display state based on the detection result.

[0081] In a fifth aspect, an embodiment of the present disclosure further provides a computer non-volatile readable storage medium, wherein a computer program is stored on the computer non-volatile readable storage medium, and when the computer program is executed by a processor, the steps of the control method of the monitoring system as described in any one of the embodiments of the first aspect above, and / or the steps of the control method of the monitoring system as described in any one of the embodiments of the second aspect above are executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] FIG1 is a flow chart of a control method of a monitoring system provided by an embodiment of the present disclosure;

[0083] FIG2 is a schematic diagram showing the effect of reasonably dividing an image to be detected according to an embodiment of the present disclosure;

[0084] FIG3 is a schematic diagram of an exemplary training process of a target neural network model provided by an embodiment of the present disclosure;

[0085] FIG4 is a schematic diagram of an exemplary network architecture of knowledge distillation provided by an embodiment of the present disclosure;

[0086] FIG5 is a schematic diagram of a network architecture of an exemplary student model provided in an embodiment of the present disclosure;

[0087] FIG6a is a diagram of an exemplary terminal workflow provided by an embodiment of the present disclosure;

[0088] FIG6 b is a flowchart of an exemplary storage and computing integrated system workflow provided by an embodiment of the present disclosure;

[0089] FIG7 is a schematic diagram of a storage and computing integrated system provided by an embodiment of the present disclosure;

[0090] FIG8 is a schematic diagram of a control device of a monitoring system provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0092] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "an" or "the" do not indicate a quantitative limitation, but rather indicate the presence of at least one. Similar words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0093] In this disclosure, "multiple or several" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0094] With the in-depth research and widespread adoption of artificial intelligence algorithms, particularly deep learning neural networks, smart electronic devices and related application scenarios have become ubiquitous, such as facial recognition, voice recognition, smart homes, security surveillance, and autonomous driving. However, due to the limitations of the classic von Neumann computing architecture, data storage and processing are separated, with computing and storage functions performed by the central processing unit (CPU) and memory, respectively. This performance gap creates a "memory wall," resulting in significant energy consumption in frequent transfers between the computing core and memory. However, current integrated computing and storage designs offer a solution to this problem. Integrated computing and storage combine memory and computing more tightly than traditional computer architectures, reducing the overhead associated with memory access and addressing the "memory wall" problem. The three key elements of artificial intelligence are computing power, data, and algorithms. Integrated computing and storage offer high computing power, low power consumption, and low latency, making them poised to play a significant role in the future of artificial intelligence. Currently, common computing and storage methods include flash memory and static random-access memory (SRAM). Generally, many national or industry standards associated with smart terminal products have certain standby power consumption requirements. Designing an intelligent wake-up function on such products to support 24-hour standby wake-up through voice, personnel detection, and other methods requires ensuring that the entire detection system has low standby power consumption to meet the corresponding energy efficiency standards.

[0095] In view of this, an embodiment of the present disclosure provides a control method for a monitoring system, specifically, obtaining an image to be detected; dividing the image to be detected into multiple target sub-images according to the shooting angle of a shooting device that shoots the image to be detected; for each of the multiple target sub-images, using a pre-trained target neural network model, processing the target sub-image to obtain a recognition result of the target sub-image; based on the recognition results corresponding to each of the target sub-images, obtaining a detection result of whether a target object exists in the image to be detected, and sending the detection result to a terminal, so that the terminal determines a display status based on at least the detection result.

[0096] The embodiment of the present disclosure uses a neural network to detect the image captured by the shooting device, and feeds back the detection result to the terminal so that the terminal can judge the current display state based on the detection result. For example, if the target object is not detected for a long time, it can enter a sleep state to save system power consumption; if after entering the sleep state, the detection result indicates that the target object appears, it can automatically start and restore normal display. In addition, the intelligent wake-up application scenario of the present disclosure, while saving system power consumption, combines the neural network image recognition technology, and reasonably divides the image to be detected captured by the shooting device into multiple target sub-images according to the shooting angle, recognizes each target sub-image separately, integrates the recognition results of each target sub-image, obtains the detection result, improves the recognition accuracy of the target object in the image to be detected, and thus more accurately controls the wake-up timing of the terminal. In addition, the storage and computing integrated module is used as a system architecture that integrates storage and computing to support complex data operations in the neural network, greatly avoiding the energy loss caused by the data transfer process and reducing system power consumption.

[0097] For ease of understanding, a control method for a monitoring system provided by an embodiment of the present disclosure is first described in detail. The execution subject of the control method of the monitoring system can be a storage-computing integrated system with certain computing capabilities, and the storage-computing integrated system includes a processing module, a storage-computing integrated module and an external storage module; wherein the processing module is, for example, the main processor CPU in the storage-computing integrated system, the storage-computing integrated module can be, for example, a storage-computing core integrated with Flash or SRAM, and the external storage module can be, for example, an external storage module DRAM, which has a faster response speed and reduces the power consumption of the monitoring system. The storage-computing integrated system can be part of the monitoring system. At the same time, the present disclosure designs a lightweight target neural network model, migrates it to the storage-computing integrated module, and adopts a low-power storage-computing integrated method to assist the target neural network in reasoning, complete the wake-up service of the intelligent terminal, reduce power consumption, and achieve a fast response speed.

[0098] FIG1 is a flow chart of a control method of a monitoring system provided by an embodiment of the present disclosure. As shown in FIG1 , the method includes steps S11 to S14, wherein:

[0099] S11. Obtain an image to be detected.

[0100] In this step, the image to be detected is an image captured by a camera and is intended to be detected for the presence of a target object. This step S11 can, for example, be the execution process of a processing module in a storage and computing system. In one case, the image to be detected is an image captured in real time by the camera; in another case, the image to be detected can be an image that is screened out from the images captured in real time by the camera by the processing module and is highly likely to contain a target object. The screening method can adopt technical solutions such as the frame difference method, which is not specifically limited in the embodiments of the present disclosure.

[0101] The shooting device can be a device with image acquisition function such as a camera. The shooting device captures images of the shooting scene in real time. The shooting scene may contain target objects, such as people moving around; however, the shooting scene may not always contain target objects in real time.

[0102] Afterwards, the acquired image to be detected can be stored in an external storage module for subsequent image segmentation process calls.

[0103] S12: Divide the image to be detected into multiple target sub-images according to the shooting angle of the shooting device that shoots the image to be detected.

[0104] This step S12 can be, for example, the execution process of a processing module in a storage-computing integrated system. The processing module reads the image to be detected from the external storage module, divides the image to be detected into multiple target sub-images based on the shooting angle of the camera used to capture the image to be detected, and stores the multiple divided target sub-images in the external storage module for subsequent invocation by the target neural network model for processing.

[0105] Since the terminal (including the display screen) in the intelligent wake-up application scenario determines the display state based on the detection results, if the confidence of the detection result is low, it will cause the terminal to mistakenly enter the sleep state, resulting in the user being unable to accurately monitor the live scene of the shooting scene. Therefore, while saving system power consumption, it is necessary to improve the accuracy of the detection results. Based on this, the processing module in the embodiment of the present disclosure divides the image to be detected into multiple target sub-images according to the shooting angle of the shooting device, and by refining the image to be detected and processing each part of the refined target sub-image separately using a pre-trained target neural network model, the image recognition accuracy can be improved.

[0106] It should be noted that the higher the degree of refinement of the image to be detected, that is, for an image to be detected, the more target sub-images are divided, the more accurate the detection result obtained by combining the recognition results of each target sub-image after processing by the target neural network model. However, the higher the degree of refinement of the image to be detected, the more model inference processes are required later, which in turn increases the power consumption of the system. Based on this, the embodiment of the present disclosure reasonably divides the image to be detected according to the shooting angle of the shooting device, which can not only ensure the image recognition accuracy, but also reasonably divide the image to be detected to avoid a large number of image detections and reduce the power consumption of the system.

[0107] Exemplarily, a plurality of shooting angles are pre-set for the shooting device, such as a normal angle, a close shot (angle smaller than the normal shooting angle), a medium shot (angle larger than the normal angle), a distant shot (angle larger than the medium shot), etc. Different shooting angles correspond to different image division methods. For example, the degree of refinement of the image to be detected corresponding to the distant shot is greater than the degree of refinement of the image to be detected corresponding to the medium shot, the degree of refinement of the image to be detected corresponding to the medium shot is greater than the degree of refinement of the image to be detected under the normal angle, and the degree of refinement of the image to be detected under the normal angle is greater than the degree of refinement of the image to be detected corresponding to the close shot.

[0108] S13. For each target sub-image among the multiple target sub-images, use a pre-trained target neural network model to process the target sub-image to obtain a recognition result of the target sub-image.

[0109] This step S13 may be, for example, the execution process of a storage-computing module in a storage-computing system. The storage-computing module integrates a target neural network model. The storage-computing module reads a target sub-image from an external storage module, processes any target sub-image using a pre-trained target neural network model, obtains a recognition result for the target sub-image, and stores the recognition result in the external storage module.

[0110] Specifically, the target sub-image is input into the target neural network model, and the target sub-image is subjected to feature extraction and analysis using multiple network layers to determine whether the target object exists. In the process of extracting and analyzing the features of the target sub-image, the operations and outputs of different network layers are processed using a memory and computing core (Flash or SRAM) architecture. For example, for a convolutional layer in a convolutional neural network, during the convolution operation, the multi-channel intermediate results corresponding to the internal convolution downsampling process are cached until the convolution output result of a target sub-image is output. This convolution output result can be used as the input of a convolutional layer, and each convolution layer is executed sequentially until the target neural network model outputs the recognition result of the target sub-image.

[0111] This integrated storage and computing approach caches the intermediate results of the internal convolution to facilitate the call and processing of the next convolution layer. Compared with the system architecture with a memory wall in traditional technology, it avoids the energy loss caused by the data transfer process, improves computing efficiency, and reduces system power consumption. It has a high application prospect for portable devices.

[0112] The target neural network model can adopt a lightweight classification model, and the result of its recognition may be the presence of the target object or the absence of the target object. Here, the target object can be, for example, a person or an object.

[0113] S14. Based on the recognition results corresponding to the target sub-images, a detection result of whether the target object exists in the image to be detected is obtained, and the detection result is sent to the terminal, so that the terminal determines a display state based on at least the detection result.

[0114] This step S14 can be, for example, executed by a processing module in a storage-computing system. The processing module obtains the recognition results of each target sub-image divided into the image to be detected from the external storage module, and based on the recognition results corresponding to each target sub-image, obtains a detection result of whether the target object exists in the image to be detected, and sends the detection result to the terminal.

[0115] Exemplarily, the processing module may obtain a detection result of "the target object exists" based on the recognition results corresponding to the target sub-images, when any recognition result indicates that the target object exists.

[0116] Exemplarily, the processing module may obtain a detection result of "target object exists" based on the recognition results corresponding to each target sub-image, if the proportion of results indicating the existence of the target object in multiple recognition results exceeds a set value among all recognition results.

[0117] The processing module sends the detection result to the terminal, and the main control module in the terminal can determine the display state based on at least the detection result; the display screen in the terminal can determine whether to display the picture normally or enter sleep mode based on the currently determined display state.

[0118] The control method of the monitoring system provided by the present invention adopts a storage and computing integrated system to realize data processing, thereby avoiding energy loss caused by the data transfer process, improving computing efficiency, and reducing system power consumption; at the same time, the image to be detected captured by the shooting device is reasonably divided into multiple target sub-images according to the shooting angle, and combined with a lightweight target neural network model, it can accurately identify whether there is a target object in the image to be detected, thereby determining a more accurate display state. When the display screen enters the sleep state, the system power consumption is saved, and the terminal can remain in standby for a long time without being actively awakened, thereby solving the problem of high power consumption and reducing a lot of maintenance costs.

[0119] In some embodiments, if the entire image to be detected is input into the model for classification and recognition, the target object is not concentrated, which will cause inaccurate recognition results. Therefore, in order to improve the recognition accuracy of the model, for step S12, the image to be detected can be reasonably divided according to the shooting angle of the shooting device. Specifically:

[0120] When the shooting angle of view is within the preset angle of view range, the image to be detected is divided into a first sub-area 01, a second sub-area 02 and a third sub-area 03 arranged side by side along the first direction Y, and the first sub-area 01 and the third sub-area 03 partially overlap with the second sub-area 02; wherein the width of the first sub-area 01 in the first direction Y is equal to the width of the third sub-area 03 in the first direction Y, and the width of the second sub-area 02 in the first direction Y is greater than the width of the first sub-area 01 in the first direction Y; the first direction Y is the height direction of the image to be detected; the part of the image to be detected located in the first sub-area 01 is divided into a plurality of first sub-images 011 arranged side by side along the second direction X, and the width of at least some of the first sub-images 011 in the second direction X is equal; the second direction X is the width direction of the image to be detected; the portion of the image to be detected located in the second sub-region 02 is divided into a plurality of second sub-images 012 arranged side by side along the second direction X, with at least some of the second sub-images 012 having equal widths in the second direction X; the portion of the image to be detected located in the third sub-region 03 is divided into a plurality of third sub-images 013 arranged side by side along the second direction X, with at least some of the third sub-images 013 having equal widths in the second direction X; the target sub-image includes a first sub-image 011, a second sub-image 012, and a third sub-image 013; the width of the third sub-image 013 in the second direction X is greater than the width of the second sub-image 012 in the second direction X, and the width of the second sub-image 012 in the second direction X is greater than the width of the first sub-image 011 in the second direction X.

[0121] In this embodiment, the preset viewing angle range can be set according to actual conditions. Figure 2 is a schematic diagram of the effect of reasonably dividing the image to be detected provided by the embodiment of the present disclosure. As shown in Figure 2, for example, the preset viewing angle range is a range within which the normal viewing angle exists. The image to be detected is divided into three areas, specifically a first sub-area 01, a second sub-area 02, and a third sub-area 03 arranged side by side along a first direction Y, respectively located at the top, middle, and bottom of the image to be detected when placed in front. The second sub-area 02 is located between the first sub-area 01 and the third sub-area 03. The positions of the deployment scenarios corresponding to the first sub-area 01, the second sub-area 02, and the third sub-area 03 are successively farther away from the camera. For example, the width of the first sub-area 01 in the first direction Y is equal to the width of the third sub-area 03 in the first direction Y, and the width of the second sub-area 02 in the first direction Y is greater than the width of the first sub-area 01 in the first direction Y. The widths of the first sub-area 01, the second sub-area 02, and the third sub-area 03 in the second direction X are equal. The second direction X is the width direction of the image to be detected.

[0122] Continuing with Figure 2, in actual application scenarios, the closer a person is to the camera, the larger the area occupied by their body; the further away they are from the camera, the smaller the area occupied by their body. Therefore, the number of first sub-images 011 located in first sub-area 01 is greater than the number of second sub-images 012 located in second sub-area 02, and the number of second sub-images 012 located in second sub-area 02 is greater than the number of third sub-images 013 located in third sub-area 03. For example, the width of third sub-image 013 in the second direction X is greater than the width of second sub-image 012 in the second direction X, and the width of second sub-image 012 in the second direction X is greater than the width of first sub-image 011 in the second direction X. At least some of the first sub-images 011 have equal widths in the second direction X. For example, among the plurality of first sub-images 011 arranged side by side along the second direction X, except for the first and last first sub-images 011 arranged side by side along the second direction X, the remaining first sub-images 011 have the same widths in the second direction X; the first and last first sub-images 011 have the same widths in the second direction X. At least some of the second sub-images 012 have equal widths in the second direction X. For example, among the plurality of second sub-images 012 arranged side by side along the second direction X, except for the first and last second sub-images 012 arranged side by side along the second direction X, the remaining second sub-images 012 have the same widths in the second direction X; the first and last second sub-images 012 have the same widths in the second direction X. At least some of the third sub-images 013 have equal widths in the second direction X. For example, the widths of all third sub-images 013 in the second direction X are equal.

[0123] Furthermore, for locations in the deployment scene that are relatively far from the camera (e.g., the location corresponding to first sub-image 011 and the location corresponding to second sub-image 012), each first sub-image 011 at least partially overlaps in the second direction X; each second sub-image 012 at least partially overlaps in the second direction X, further resolving the issue of inaccurate recognition at the splicing of adjacent target sub-images and improving the recognition accuracy of target sub-images. For locations in the deployment scene that are relatively close to the camera (e.g., the location corresponding to third sub-image 013), to simplify the image segmentation method, each third sub-image 013 can be set to have no overlap in the second direction X. Since target objects closer to the camera occupy a larger portion of the image than target objects farther away, the lack of overlap between adjacent third sub-images 013 in the second direction X does not affect the recognition accuracy of third sub-image 013, or the impact is minimal and negligible.

[0124] Furthermore, the width ratio of the second sub-area 02 to the first sub-area 01 in the first direction Y is 2:1; the width ratio of the third sub-image 013 to the second sub-image 012 in the second direction X is 3:2, and the width ratio of the second sub-image 012 to the first sub-image 011 in the second direction X is 4:3.

[0125] Of course, in combination with different application scenarios and the size of the image captured by the camera, different sub-areas can be divided, and different width ratios can also be set. This disclosure does not specifically limit this.

[0126] Furthermore, among the multiple first sub-images 011 arranged side by side along the second direction X, except for the first first sub-image 011 and the last first sub-image 011 arranged side by side along the second direction X, the widths of the remaining first sub-images 011 in the second direction X are the same; the widths of the first first sub-image 011 and the last first sub-image 011 in the second direction X are the same and are smaller than the widths of the remaining first sub-images 011 in the second direction X; the ratio of the overlapping width of two adjacent first sub-images 011 in the second direction X to the widths of the remaining first sub-images 011 in the second direction X is 1:10; among the multiple second sub-images 012 arranged side by side along the second direction X, except for the first second sub-image 012 and the last second sub-image 012 arranged side by side along the second direction X, the widths of the remaining second sub-images 012 in the second direction X are the same; the widths of the first second sub-image 012 and the last second sub-image 012 in the second direction X are the same and are smaller than the widths of the remaining second sub-images 012 in the second direction X; the overlapping widths of two adjacent second sub-images 011 in the second direction X are 1:10. The ratio of the overlapping width of the first sub-image 011 and the second sub-image 012 in the second direction X to the width of the remaining second sub-images 012 in the second direction X is 1:10. Among the multiple third sub-images 013 arranged side by side along the second direction X, except for the first third sub-image 013 and the last third sub-image 013 arranged side by side along the second direction X, the widths of the remaining third sub-images 013 in the second direction X are the same. The widths of the first third sub-image 013 and the last third sub-image 013 in the second direction X are the same and smaller than the widths of the remaining third sub-images in the second direction X. The ratio of the overlapping width of two adjacent third sub-images 013 in the second direction X to the widths of the remaining third sub-images 013 in the second direction X is 1:10. The ratio of the overlapping width of the first sub-image 011 and the second sub-image 012 adjacent in the first direction Y in the first direction Y to the width of the second sub-image 012 in the first direction Y is 1:10. The ratio of the overlapping width of the third sub-image 013 and the second sub-image 012 adjacent in the first direction Y in the first direction Y to the width of the second sub-image 012 in the first direction Y is 1:10.

[0127] In combination with the above embodiments, under a normal viewing angle, the image to be detected is divided into multiple target sub-images through the above division method, specifically including multiple first sub-images 011, multiple second sub-images 012 and multiple third sub-images 013, which refines the image to be detected and improves the accuracy of subsequent detection results while controlling the power consumption of the system within a reasonable range.

[0128] In some embodiments, for step S11, a frame difference method is used to detect whether there is a moving target at the shooting scene of the shooting device, and the moving target can be considered as a moving object or person. Specifically, the processing module obtains multiple frames of continuous video frames shot by the shooting device; based on the multiple frames of continuous video frames, the frame difference method is used to determine whether there is a moving target at the shooting scene of the shooting device, and the judgment result is stored in the external storage module. For example, if there is a moving target at the shooting scene, the position of the image of the target is different in different video frames. The frame difference algorithm performs pixel value difference operations on two or three consecutive video frames in time, subtracts the pixel values ​​corresponding to different frames, and determines the absolute value of the grayscale value. When the absolute value exceeds a certain threshold, it can be determined that there is a moving target. If there is a moving target, the video frame shot by the shooting device is used as the image to be detected.

[0129] In this embodiment, before using the model for detection, the frame difference method is used to preliminarily screen the video frames to find the video frames to be detected in which the target object is likely to exist, thereby simplifying the subsequent model processing and improving the detection efficiency.

[0130] In some embodiments, FIG3 is a schematic diagram of an exemplary training process of a target neural network model provided by an embodiment of the present disclosure. As shown in FIG3 , the processing module includes a training process of executing steps S31 to S34, wherein:

[0131] S31. Obtain a first training data set and a second training data set.

[0132] Here, the first training data set can be a labeled public data set and / or collected video data. Alternatively, the first training data set can also be a data set that further enriches the sample types based on the labeled public data set and / or collected video data, so that the resulting training images are more diverse. The method of enriching the sample types can be referred to in the following steps S311 to S315, and the detailed process is not described here. Among them, the first training data set includes multiple training images, and each training image includes a corresponding sample label.

[0133] It should be noted that in order to improve the accuracy of model training, a large amount of sample data support is required. However, sample production is not easy. Therefore, in order to obtain a large number of samples in the early stage, this embodiment obtains a first training data set with noise as the training set of the teacher machine learning model.

[0134] Here, the second training dataset can be obtained by filtering the first training dataset, that is, a noise-free dataset that has been carefully selected and accurately classified. The second training dataset includes a portion of the training images in the first training dataset. The second training dataset includes multiple training images, each of which includes a corresponding sample label.

[0135] In some embodiments, the first training data set includes noisy training images. For example, positive samples are preset as images with human head features, and negative samples are preset as images without human head features (e.g., images with no human features at all, and images with partial human limb features (excluding head features)). The training images in the first training data set that meet the characteristics of the target object are marked as positive sample labels; those that do not meet the characteristics of the target object are marked as negative sample labels. For example, a training image in the first training data set contains human features, so the training image is marked as a positive sample image; however, the training image does not contain head features, so the training image contains noise.

[0136] Manual screening can be used to select accurately classified training images from the noisy first training dataset as samples in the second training dataset. For example, training images containing pedestrian features above the shoulders, including head features, can be used as positive samples in the second training dataset, while training images containing other limb features of pedestrians (without head features) can be used as negative samples in the second training dataset. Alternatively, traditional image recognition technology can be used to detect whether the training images in the first training dataset contain features above the shoulders, including head features. If so, the training image is selected as a positive sample in the second training dataset; if not, the training image is selected as a negative sample in the second training dataset.

[0137] S32. Train the teacher machine learning model to be trained according to the first training data set to obtain a preliminarily trained teacher machine learning model.

[0138] Exemplarily, a training image A in a first training dataset is input into the teacher machine learning model to be trained, and an output result A corresponding to the training image A is obtained. The output result A is a result indicating whether the target object exists in the training image A. If the sample label of the training image A indicates the presence of the target object, a third loss function is determined based on the output result A and the sample label of the training image A. The above process is repeated, traversing the training images in the first training dataset in sequence, and the third loss function is determined to continuously train the teacher machine learning model to be trained, ultimately obtaining a preliminarily trained teacher machine learning model.

[0139] S33. Train the preliminarily trained teacher machine learning model according to the second training data set to obtain a trained teacher machine learning model.

[0140] Exemplarily, a training image B in the second training dataset is input into the preliminarily trained teacher machine learning model to obtain an output result B corresponding to the training image B, where the output result B indicates whether the target object exists in the training image B. If the sample label of the training image B indicates the presence of the target object, a fourth loss function is determined based on the second output result and the sample label of the training image B. The above process is repeated, sequentially traversing the training images in the second training dataset, and the fourth loss function is determined to continuously train the preliminarily trained teacher machine learning model, ultimately obtaining a fully trained teacher machine learning model.

[0141] S34. Based on the second training data set and the trained teacher machine learning model, the knowledge distillation training method is used to train the student machine learning model to be trained, and the trained student machine learning model is obtained as the target neural network model.

[0142] Here, the trained teacher machine learning model is the teacher model in the knowledge distillation process; the student machine learning model to be trained is the student model in the knowledge distillation process.

[0143] Figure 4 is a schematic diagram of an exemplary network architecture of knowledge distillation provided by an embodiment of the present disclosure. As shown in Figure 4, it includes a teacher model and a student model. The teacher model includes m network layers; the student model includes n network layers; here, the teacher model and the student model can be homogeneous networks or heterogeneous networks.

[0144] The process of training the student model also uses the hard labels in the second training dataset (i.e., the sample labels carried by the training images) to train the teacher model, and then uses the soft labels obtained in the teacher model (i.e., the output of the softmax layer in the teacher model) combined with the hard labels in the second training dataset to jointly determine the loss function of the student model.

[0145] Continuing with Figure 4, the output of the teacher model's mth layer is processed by the softmax layer to obtain the first output. The output of the student model's nth layer is also processed by the softmax layer to obtain the second output. The loss function is constructed based on the first and second outputs and the sample labels.

[0146] This embodiment uses the knowledge distillation method to convert a larger teacher model into a smaller student model while retaining performance close to that of the teacher model, thereby solving the problem of insufficient hardware for deploying the target neural network model at the edge segment.

[0147] In some embodiments, due to the hardware limitations of the integrated storage and computing chip in the intelligent wake-up application scenario of the present disclosure, it is necessary to design a lightweight target neural network model to complete the target detection task. The target neural network model architecture is also the network architecture of the student model.

[0148] FIG5 is a schematic diagram of the network architecture of an exemplary student model provided by an embodiment of the present disclosure. As shown in FIG5 , it includes five convolutional unit blocks and a fully connected layer fc; each convolutional unit block includes two convolutional layers. For example, for the two convolutional layers in the first convolutional unit block1, the second convolutional unit block2, and the third convolutional unit block3, the first convolutional layer (that is, the convolutional layer that first processes data) uses a convolution kernel of size 3×3 for convolution, a convolution step of 2, and a padding value of 1; the second convolutional layer (that is, the convolution layer that receives the output data of the previous convolutional layer) uses a convolution kernel of size 3×3 for convolution, a convolution step of 1, and a padding value of 1. For the two convolutional layers in the fourth convolutional unit block4 and the fifth convolutional unit block5, the front convolutional layer (that is, the convolutional layer that processes data first) uses a 5×5 convolution kernel for convolution, a convolution step of 2, and a padding value of 1; the back convolutional layer (that is, the convolutional layer that receives the output data of the front convolutional layer) uses a 3×3 convolution kernel for convolution, a convolution step of 1, and a padding value of 1.

[0149] The above-mentioned target neural network model meets the lightweight requirements and is migrated to the storage and computing integrated chip. During its application process, it has faster response speed and lower power consumption.

[0150] In some embodiments, the second training dataset includes a plurality of training images annotated with sample labels.

[0151] As shown in FIG4 , for step S34 , the specific process of training to obtain the target neural network model includes the following steps S341 to S346 , wherein:

[0152] S341. Input the training image into the trained teacher machine learning model to obtain the first output result of the trained teacher machine learning model.

[0153] Exemplarily, the training image B in the second training data set is input into the teacher machine learning model to be trained to obtain a first output result corresponding to the training image B. The first output result is the result output by the teacher machine learning model to be trained indicating whether the target object exists in the training image B.

[0154] S342. Input the training image into the student machine learning model to be trained to obtain a second output result of the student machine learning model to be trained.

[0155] Continuing with the above example, the training image B in the second training data set is input into the student machine learning model to be trained to obtain the second output result corresponding to the training image B. The second output result is the result output by the student machine learning model to be trained indicating whether the target object exists in the training image B.

[0156] S343. Determine a first loss function according to the first output result and the second output result.

[0157] Following steps S341 and S342, the first loss function can be determined based on the mean square error between the first output result and the second output result, or based on the first output result and the second output result using the KLD loss function algorithm (ie, KLDloss).

[0158] S344. Determine a second loss value based on the second output result and the sample label of the training image.

[0159] The second loss function can be determined based on the overlap between the second output result (i.e., the predicted label of the training image B predicted by the student machine learning model to be trained) and the sample label of the training image (i.e., the real hard label).

[0160] S345. Obtain a weighted loss function according to the first loss function and the second loss function.

[0161] The first loss function and the second loss function are weighted according to a preset weight to obtain a weighted loss function. Here, the preset weight can be determined based on experience and is not specifically limited in the embodiment of the present disclosure.

[0162] Obtain the weighted coefficients of the first loss function and the second loss function. In this embodiment, since the weighted loss function is composed of the first loss function and the second loss function, by determining one of the weighted coefficients (α), the other weighted coefficient (i.e., 1-α) can be determined. Afterwards, the first loss function L1 and the second loss function L2 are weighted and summed according to the weighted coefficients to determine the weighted loss function. The weighted loss function L, combined with the above description, can be specifically expressed as: L = α × L1 + (1-α) × L2.

[0163] S346. Adjust the parameters of the student machine learning model to be trained according to the weighted loss value until the weighted loss value converges, and obtain the trained student machine learning model as the target neural network model.

[0164] In some embodiments, the first training dataset may be a dataset that is further enriched with sample types based on the original dataset. The specific processing module includes the steps of generating diversified training images, specifically referring to the following steps S311 to S317, wherein:

[0165] S311. Obtain the original data set.

[0166] The original dataset includes multiple initial sample images.

[0167] For example, the original dataset may be a public dataset that has been annotated and / or collected video data. "Annotation" here refers to annotating the first reference frame of the target object in the initial sample image.

[0168] The initial sample images in the original dataset may or may not contain the target object. For those initial sample images that do not contain the target object (meaning that there are no features of the target object at all), the first reference frame does not exist, or the position information of the first reference frame is empty.

[0169] S312: Identify the target object on the initial sample image. If the target object exists, execute step S313; if not, execute step S317.

[0170] For example, the target detection network retinaface or the neural network yolov5 model can be used to identify the target object in the initial sample image.

[0171] Here, the initial sample image without the target object, i.e., the initial sample image without any target object features, is labeled as a negative sample, and the initial sample image labeled with the negative sample label is used as the negative sample (training image) in the first training data set.

[0172] It should be noted that for an initial sample image containing a target object, the following steps S313 to S316 are included.

[0173] S313: Determine a first reference frame containing the target object.

[0174] When it is determined that the target object exists in the initial sample image, a first reference frame is marked for the area where the target object is located, so that the first reference frame frames the target object. Position information of the first reference frame indicates the position of the target object in the initial sample image.

[0175] S314 : Update the position of the first reference frame according to the position information of the first reference frame to obtain a second reference frame.

[0176] The specific updating method may include: randomly moving a specific coordinate point, randomly scaling, randomly cropping, etc.

[0177] Example 1: For randomly moving detection points, the specific process includes: based on the position information of the first reference frame (that is, the position coordinates in the initial sample image), the coordinates of the specific coordinate points pre-set in the first reference frame can be randomly moved to obtain new contour coordinate points to form a second reference frame.

[0178] Example 2: For random scaling, the specific process includes: based on the position information of the first reference frame (that is, the position coordinates in the initial sample image), the center point coordinates of the first reference frame can be kept unchanged, and the width and height dimensions of the first reference frame can be randomly scaled to obtain new contour coordinate points to form a second reference frame.

[0179] Example 3: For random cropping, the specific process includes: determining a new center point according to the position information of the first reference frame (that is, the position coordinates in the initial sample image) and the first preset center point range; keeping the new center point coordinates unchanged, randomly adjusting the width and height of the first reference frame according to the preset cropping range to obtain new contour coordinate points and form a second reference frame.

[0180] In other embodiments, the position of the first reference frame is updated, and the specific updating method can be combined with the above examples. For example, Example 1 and Example 2 are executed sequentially, specifically: based on the position information of the first reference frame, the coordinates of the specific coordinate points pre-set in the first reference frame can be randomly moved to obtain new contour coordinate points to form a transition frame; based on the position information of the transition frame, the coordinates of the center point of the transition frame are kept unchanged, and the width and height of the transition frame are randomly scaled to obtain new contour coordinate points to form a second reference frame. For another example, Example 1 and Example 3 are executed sequentially, specifically: based on the position information of the first reference frame, the coordinates of the specific coordinate points pre-set in the first reference frame can be randomly moved to obtain new contour coordinate points to form a transition frame; based on the position information of the transition frame, a new center point is determined according to a preset center point range; and the new center point coordinates are kept unchanged, and the width and height of the transition frame are randomly adjusted according to a preset cropping range to obtain new contour coordinate points to form a second reference frame. For another example, Example 2 and Example 3 are executed sequentially. Specifically: based on the position information of the first reference frame, the center point coordinates of the first reference frame can be kept unchanged, and the width and height dimensions of the first reference frame can be randomly scaled to obtain new contour coordinate points to form a transition frame; based on the position information of the transition frame, a new center point can be determined according to a preset center point range; and the new center point coordinates can be kept unchanged, and the width and height dimensions of the transition frame can be randomly adjusted according to a preset cropping range to obtain new contour coordinate points to form a second reference frame.

[0181] In other embodiments, the position of the first reference frame is updated. The specific updating method may be a combination of all the above examples (Example 1, Example 2, and Example 3), including the following steps S314-1 to S314-3, wherein:

[0182] S314 - 1 . According to the position information of the first reference frame, move a specific coordinate point in the first reference frame to obtain a third reference frame.

[0183] Here, the specific coordinate points may be pre-set vertices of the outer contour of the first reference frame, such as vertex 1 at the upper left corner, vertex 2 at the upper right corner, vertex 3 at the lower left corner, and vertex 4 at the lower right corner. In this embodiment, vertex 1 at the upper left corner and vertex 4 at the lower right corner may be selected as specific coordinate points for performing diversified updating of the reference frame.

[0184] For example, the coordinates of the vertex 1 in the upper left corner are (lt x ,lt y ), the coordinates of vertex 4 in the lower right corner are (rb x ,rb y ); Let the coordinates of vertex 1 (lt x ,lt y ) has a random moving value of δ x , δ y; The coordinates of vertex 4 (rb x ,rb y )’s random moving value σ x , σ y , the value can be positive or negative. After random movement, the coordinates of the new vertex 1 are (lt1 x ,lt1 y ), that is: lt1 x =lt x +δ x lt1 y =lt y +δ y

[0185] The coordinates of the new vertex 4 are (rb1 x ,rb1 y ), that is: rb1 x =rb x +σ x rb1 y =rb y +σ y

[0186] S314 - 2 . Based on the position information of the third reference frame and with the center point of the third reference frame as the center, the width and height of the third reference frame are adjusted according to a preset scaling factor to obtain a fourth reference frame.

[0187] Continuing with the above example, the coordinates of the center point of the third reference frame are (c x ,c y ), the width dimension of the third reference frame is w, and the height dimension is h; the preset scaling factor includes the width scaling factor s x and height scaling factor s y According to the preset scaling factor, adjust the width and height of the third reference frame to obtain the coordinates of vertex 1 in the upper left corner of the fourth reference frame (lt2 x ,lt2 y ), the coordinates of vertex 4 at the lower right corner of the fourth reference frame (rb2 x ,rb2 y ), where: c x =(lt1 x +rb1 x ) / 2, c y =(lt1 y +rb1 y ) / 2 w=(rb1 x -lt1 x ), h=(rb1 y -lt1 y ) lt2 x =c x -sx ×w / 2 lt2 y =c y -s y ×h / 2 rb2 x =c x +s x ×w / 2 rb2 y =c y +s y ×h / 2

[0188] S314-3. Determine a new center point based on the position information of the fourth reference frame and the first preset center point range; and adjust the width and height of the fourth reference frame based on the new center point and the preset cropping range to obtain a second reference frame.

[0189] Here, the width of the fourth reference frame is w'=(rb2 x -lt2 x ), high h'=(rb2 y -lt2 y ); The first preset center point range is also the randomly generated center point (cnew x ,cnew y ) in the range where cnew x ∈[0.45×w', 0.55×w'], cnew y ∈[0.45×h',0.55×h']. The preset cropping range is the range of the randomly generated width size wnew and height size hnew of the fourth reference frame, where wnew∈[w'×0.9,w'] and hnew∈[h'×0.9,h'].

[0190] According to the new center point (cnew x ,cnew y ) and the preset cropping range, adjust the width and height of the fourth reference frame to obtain the new width wnew' and height hnew', and then obtain the new contour coordinate point, the upper left corner coordinate (lt3 x ,lt3 y ), the lower right corner coordinate (rb3 x ,rb3 y ).

[0191] The above updating method can obtain diversified training images and improve the accuracy of subsequent model training.

[0192] S315 : Determine the degree of overlap between the second reference frame and the first reference frame based on the position information of the second reference frame and the position information of the first reference frame.

[0193] The IOU (Integrated Underlying Unit) represents the degree of overlap between the first and second reference frames. A larger IOU value indicates a greater overlap between the second and first reference frames, and a greater proportion of the object within the second reference frame. Conversely, a smaller IOU value indicates a greater overlap between the first and second reference frames.

[0194] S316 , marking sample labels for the portion of the sample images of the initial sample image located in the second reference frame according to a comparison result between the overlap degree and the first preset threshold, and using the portion of the sample images marked with sample labels as training images in the first training data set.

[0195] Here, the first preset threshold can be set based on experience. When the overlap is greater than or equal to the first preset threshold, the corresponding portion of the sample image is determined to be a positive sample and labeled as a positive sample. It should be noted that although the overlap is greater than or equal to the first preset threshold, the overlap only reflects the presence of a portion of the target object in the second reference frame. The features of this portion of the target object may be features that clearly indicate the user's identity, such as the head, or features that do not clearly indicate the user's identity, such as limbs.

[0196] Generating diverse training images can increase the richness of the first training dataset, thereby improving the accuracy of model training. Diversification refers to the diversification of the features of the target objects segmented in the training images, such as a large number of training images containing head features of the target object, some training images containing only limb features, some training images containing torso features, some training images containing full-body features of the target object, and so on. In some embodiments, following step S315, each initial sample image corresponds to its own degree of overlap (or each training image corresponds to its own degree of overlap).

[0197] In specific implementation, when the degree of overlap is greater than or equal to the first preset threshold, the first preset range can be generated according to the first preset probability, the second preset range can be generated according to the second preset probability, the third preset range can be generated according to the third preset probability, and the fourth preset range can be generated according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1.

[0198] When the degree of overlap is within a first preset range, a portion of the sample images in the initial sample image located in the second reference frame is determined and marked as positive sample labels, so that the ratio between the number of training images with an overlap degree within the first preset range and the total number of training images with positive sample labels in the first training data set is a first preset probability.

[0199] When the degree of overlap is within a second preset range, a portion of the sample images in the initial sample image located in the second reference frame is determined and marked as positive sample labels, so that the ratio between the number of training images with an overlap degree within the second preset range and the total number of training images with positive sample labels in the first training data set is a second preset probability.

[0200] When the overlap degree is within a third preset range, part of the sample images in the initial sample image located in the second reference frame is determined and marked as positive sample labels, so that the ratio between the number of training images with an overlap degree within the third preset range and the total number of training images with positive sample labels in the first training data set is a third preset probability.

[0201] When the degree of overlap is within a fourth preset range, a portion of the sample images in the initial sample image located in the second reference frame is determined and marked as positive sample labels, so that the ratio between the number of training images with an overlap degree within the fourth preset range and the total number of training images with positive sample labels in the first training data set is a fourth preset probability.

[0202] The overlap within the first preset range is smaller than the overlap within the second preset range, the overlap within the second preset range is smaller than the overlap within the third preset range, and the overlap within the third preset range is smaller than the overlap within the fourth preset range.

[0203] Exemplarily, the first preset threshold is 0.6; if the overlap is greater than or equal to 0.6, the first preset range 0.6~0.7 (excluding 0.7) is selected according to a probability of 50% (first preset probability), 0.7~0.8 (excluding 0.8) is selected according to a probability of 30% (second preset probability), 0.8~0.9 (excluding 0.9) is selected according to a probability of 10% (third preset probability), and 0.9~1.0 is selected according to a probability of 10% (fourth preset probability). When the degree of overlap is within the first preset range, part of the sample images located in the second reference frame are marked as positive sample labels, so that the training images with an overlap degree within the first preset range account for 50% of the total number of training images with positive sample labels; when the degree of overlap is within the second preset range, part of the sample images located in the second reference frame are marked as positive sample labels, so that the training images with an overlap degree within the second preset range account for 30% of the total number of training images with positive sample labels; when the degree of overlap is within the third preset range, part of the sample images located in the second reference frame are marked as positive sample labels, so that the training images with an overlap degree within the third preset range account for 10% of the total number of training images with positive sample labels; when the degree of overlap is within the fourth preset range, part of the sample images located in the second reference frame are marked as positive sample labels, so that the training images with an overlap degree within the fourth preset range account for 10% of the total number of training images with positive sample labels.

[0204] In this embodiment, the proportion of positive samples in the first training data set is pre-set. The first preset range is selected as 0.6-0.7 with a probability of 50%, that is, the positive samples with IOU in [0.6, 0.7) are controlled to account for 50% of the total positive sample data; the second preset range is selected as 0.7-0.8 with a probability of 30%, that is, the positive samples with IOU in [0.7, 0.8) are controlled to account for 30% of the total positive sample data; the third preset range is selected as 0.8-0.9 with a probability of 10%, that is, the positive samples with IOU in [0.8, 0.9) are controlled to account for 10% of the total positive sample data; the fourth preset range is selected as 0.9-1.0 with a probability of 10%, that is, the positive samples with IOU in [0.9, 1.0] are controlled to account for 10% of the total positive sample data. Since the target neural network model in the embodiment of the present disclosure adopts a small model and multiple target sub-images after the image to be detected is segmented, the training image will also use the segmented image of the entire image to improve the model training accuracy. In the process of generating the first training data set, as many images as possible (ie, partial sample images) with the target object segmented are generated as training images.

[0205] Of course, the above division of the proportion of positive samples in the first training data set is only one embodiment, and other proportions can be set as long as the conditions for enriching the training samples are met, which is not limited in the embodiments of the present disclosure.

[0206] S317 , marking the initial sample image with a negative sample label, and using the initial sample image marked with the negative sample label as a training image in the first training data set.

[0207] This step is to mark the negative sample labels for the initial sample images that do not contain the target object.

[0208] In some embodiments, when the overlap between the first reference frame and the second reference frame is less than a third preset threshold, the portion of the sample image where the initial sample image is located within the second reference frame is labeled as a negative sample. The second preset threshold can be set empirically and is less than the first preset threshold. For example, the third preset threshold is 0.1. Therefore, the portion of the sample image labeled as a negative sample does not necessarily lack the target object's features; rather, the features are minimal and negligible, and thus can be used as negative samples.

[0209] Of course, in addition to the above-mentioned method of producing negative samples, it is also possible to cut out a portion of the sample image that does not contain the target object from the initial sample image as a negative sample; and / or, cut out a portion of the sample image that only contains a small part of the target object features from the initial sample image as a negative sample, and so on. The embodiments of the present disclosure do not impose specific restrictions on this.

[0210] In some embodiments, after executing S312, another method of producing negative samples is also included, specifically: according to the size of the initial sample image and according to the second preset center point range, a target center point is determined in the initial sample image, and a fifth reference frame is determined with the target center point as the center; according to the position information of the fifth reference frame and the position information of the first reference frame, the overlap degree of the fifth reference frame and any first reference frame in the initial sample image is determined; if the overlap degree of the fifth reference frame and any first reference frame in the initial sample image is less than or equal to the second preset threshold, the part of the sample image of the initial sample image located in the fifth reference frame is marked as a negative sample label, and the part of the sample image marked with the negative sample label is used as the training image in the first training data set.

[0211] Here, the second preset threshold can be selected within the range of 0 to 0.1; the fifth reference frame is defined by randomly setting the target center point on the initial sample image; since part of the sample image corresponding to the first reference frame contains the target object, if the overlap between the fifth reference frame and any first reference frame is less than or equal to the second preset threshold, it means that the overlap between the fifth reference frame and any first reference frame is very small, or even 0, then part of the sample image corresponding to the fifth reference frame can be considered to have no target object, and therefore part of the sample image at the corresponding position of the fifth reference frame can be used as a negative sample training image.

[0212] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0213] It should be noted that, in addition to detecting target objects and waking up devices for operation as mentioned in the embodiments of this disclosure, portable smart terminal application services in smart homes and wearable devices can also adopt this high-efficiency storage and computing approach to reduce significant costs. Furthermore, with the development of storage and computing technology, it can support larger deep learning models and more extensive and universal services. In addition, because different storage and computing platforms support different functions and operators, this disclosure can design and build a network based on the actual hardware platform and is not limited to the target neural network model proposed in this disclosure.

[0214] In addition, the disclosed embodiment also provides another control method for a monitoring system. The execution subject of the control method for the monitoring system can be a monitoring system, which includes a storage and computing system and a terminal. The storage and computing system includes a processing module, a storage and computing module, and an external storage module. The processing module is, for example, the main processor CPU in the storage and computing system, the storage and computing module can be, for example, a storage and computing core integrated with Flash or SRAM, and the external storage module can be, for example, an external storage module DRAM, which has a faster response speed and reduces the power consumption of the monitoring system. The control method for the monitoring system includes:

[0215] The processing module obtains an image to be inspected; based on the viewing angle of the camera used to capture the image, the image to be inspected is divided into multiple target sub-images, and the sub-images are stored in an external storage module. It should be noted that the specific execution process of the image division module in the disclosed embodiment can be found in steps S11 and S12 of the control method of the monitoring system described above, and the repeated parts are not repeated here.

[0216] For each target sub-image in the multiple target sub-images, the storage and computing module uses a pre-trained target neural network model to process the target sub-image, obtain the recognition result of the target sub-image, and store the recognition result in an external storage module. It should be noted that the specific execution process of the storage and computing module in the embodiment of the present disclosure can be referred to step S13 in the control method of the monitoring system above, and the repeated parts will not be repeated. The storage and computing module is integrated with the target neural network model.

[0217] Based on the recognition results corresponding to each target sub-image, the processing module determines whether the target object exists in the image to be detected and transmits the detection result to the terminal. It should be noted that the specific execution process of the processing module in the embodiment of the present disclosure can be found in step S13 of the control method of the monitoring system described above, and the repeated parts will not be repeated here.

[0218] The terminal determines the display status based on the detection result.

[0219] The embodiment of the present disclosure provides a control method for a monitoring system, which uses a neural network to detect the image captured by the shooting device and feeds back the detection result to the terminal; the terminal determines the current display state based on the detection result. For example, if the target object is not detected for a long time, it can enter a sleep state to save system power consumption; if after entering the sleep state, the detection result indicates that the target object appears, it can automatically start and restore normal display. In addition, the intelligent wake-up application scenario of the present disclosure, while saving system power consumption, combines the neural network image recognition technology, and reasonably divides the image to be detected captured by the shooting device into multiple target sub-images according to the shooting angle, recognizes each target sub-image separately, integrates the recognition results of each target sub-image, obtains the detection result, improves the recognition accuracy of the target object in the image to be detected, and thus more accurately controls the wake-up timing of the terminal. In addition, the storage and computing integrated module is used as a system architecture that integrates storage and computing to support complex data operations in the neural network, greatly avoiding the energy loss caused by the data transfer process and reducing system power consumption.

[0220] In some embodiments, the terminal includes a display module and a main control module; the terminal determines the display state based on the detection result, including: when the main control module determines that the detection result indicates that there is a target object in the image to be detected, the main control module sends a wake-up request to the display module and resets the sleep timer to respond to the terminal's request to receive the detection result sent by the processing module; when it is determined that the detection result indicates that there is no target object in the image to be detected, and the current system time is greater than the preset time from the last time the display module was awakened, the main control module sends a sleep request to the display module and controls itself to enter a sleep state; the display module responds to the wake-up request and determines that the display state is to normally display the shooting picture of the shooting device; and responds to the sleep request and determines that the display state is to sleep.

[0221] Here, the preset duration can be set based on experience. It should be noted that a reasonable preset duration is set so that the terminal enters the sleep state appropriately in actual application scenarios. If the preset duration is too short, it will lead to frequent switching between sleep and wake-up states, which is not conducive to reducing power consumption. If the preset duration is too long, it will not be able to maximize the reduction of terminal power consumption.

[0222] Here, the display module may be, for example, a display screen, and the main control module may be, for example, a system on chip (SOC). The main control module and the image segmentation module may communicate using a protocol such as UART.

[0223] Figure 6a is an exemplary terminal workflow diagram provided by an embodiment of the present disclosure. As shown in Figure 6a, the main control module: S61, starts system initialization; S62, sends a startup instruction to the image segmentation module, and receives a response confirmation from the image segmentation module, and executes S63 in sequence; S63, resets the sleep timer; S64, polls the serial port to see if the target object is recognized, and sends a response confirmation to the image segmentation module; if so, returns to execute step S63; if not, executes S65; S65, determines whether the preset time is exceeded; if so, executes S66; if not, returns to execute S64; S66, the main control SOC enters the sleep state, and sends a sleep request to the display module, and executes S67 in sequence; S67, polls the serial port to see if the target object is recognized, and sends a response confirmation to the image segmentation module; if so, executes S68; if not, repeats S67; S68, wakes up the main control SOC, sends a wake-up request to the display module, and returns to step S63.

[0224] FIG6b is a flowchart of an exemplary storage and computing system workflow provided by an embodiment of the present disclosure. As shown in FIG6b , the storage and computing system: S601, system initialization begins; S602, the processing module polls the serial port to identify the startup command sent by the main control module, and sends a response confirmation to the main control module, and then executes S603 in sequence; S603, the processing module detects moving targets, that is, uses the frame difference method to determine whether there is a moving target in the shooting scene of the shooting device; if so, executes step 604, that is, starts the storage and computing module to perform the target object recognition function. For detailed process, refer to the process of determining the detection result in the above steps S12 to S14; if not, repeats step 603. S604, detects whether there is a target object, and sends the detection result to the terminal.

[0225] In some embodiments, the monitoring system further includes a shooting device, and the image segmentation module can obtain data from the low-power shooting device through protocols such as SPI.

[0226] The following embodiment is an explanation of the control process of the monitoring system implemented by the storage and computing integrated system. For detailed process, please refer to the description of the specific embodiment of the control method of the monitoring system with the storage and computing integrated system as the execution body. The repeated parts will not be repeated.

[0227] In some embodiments, the processing module divides the image to be detected into multiple target sub-images according to the shooting angle of the shooting device, including:

[0228] When the shooting angle of view is within a preset angle of view range, the image to be detected is divided into a first sub-region, a second sub-region, and a third sub-region arranged in sequence along a first direction, and the first sub-region and the third sub-region partially overlap with the second sub-region; wherein the width of the first sub-region in the first direction is equal to the width of the third sub-region in the first direction, and the width of the second sub-region in the first direction is greater than the width of the first sub-region in the first direction; and the first direction is the height direction of the image to be detected;

[0229] Dividing the portion of the image to be detected located in the first sub-area into a plurality of first sub-images arranged side by side along a second direction, wherein at least some of the first sub-images have equal widths in the second direction; the second direction being the width direction of the image to be detected;

[0230] Dividing the portion of the image to be detected located in the second sub-area into a plurality of second sub-images arranged side by side along a second direction, wherein at least some of the second sub-images have equal widths in the second direction;

[0231] Dividing the portion of the image to be detected located in the third sub-region into a plurality of third sub-images arranged side by side along the second direction, wherein at least some of the third sub-images have equal widths in the second direction; the target sub-image includes the first sub-image, the second sub-image, and the third sub-image;

[0232] The width of the third sub-image in the second direction is greater than the width of the second sub-image in the second direction, and the width of the second sub-image in the second direction is greater than the width of the first sub-image in the second direction.

[0233] In some embodiments, the first sub-images at least partially overlap in the second direction; the second sub-images at least partially overlap in the second direction; and the third sub-images have no overlap in the second direction.

[0234] In some embodiments, the ratio of the width of the second sub-region to the width of the first sub-region in the first direction is 2:1;

[0235] The ratio of the widths of the third sub-image to the second sub-image in the second direction is 3:2, and the ratio of the widths of the second sub-image to the first sub-image in the second direction is 4:3.

[0236] In some embodiments, among the plurality of first sub-images arranged side by side along the second direction, except for the first first sub-image and the last first sub-image arranged side by side along the second direction, the widths of the remaining first sub-images in the second direction are the same; the widths of the first first sub-image and the last first sub-image in the second direction are the same and smaller than the widths of the remaining first sub-images in the second direction; and the ratio of the overlapping width of two adjacent first sub-images in the second direction to the widths of the remaining first sub-images in the second direction is 1:10;

[0237] Among the plurality of second sub-images arranged side by side along the second direction, except for the first second sub-image and the last second sub-image arranged side by side along the second direction, the remaining second sub-images have the same width in the second direction; the first second sub-image and the last second sub-image have the same width in the second direction and are smaller than the widths of the remaining second sub-images in the second direction; and the ratio of the overlapping width of two adjacent second sub-images in the second direction to the widths of the remaining second sub-images in the second direction is 1:10;

[0238] Among the plurality of third sub-images arranged side by side along the second direction, except for the first and last third sub-images arranged side by side along the second direction, the remaining third sub-images have the same width in the second direction; the first and last third sub-images have the same width in the second direction and are smaller than the widths of the remaining third sub-images in the second direction; and the ratio of the overlapping width of two adjacent third sub-images in the second direction to the widths of the remaining third sub-images in the second direction is 1:10;

[0239] For the first sub-image and the second sub-image adjacent in the first direction, the ratio of the overlapping width in the first direction to the width of the second sub-image in the first direction is 1:10; for the third sub-image and the second sub-image adjacent in the first direction, the ratio of the overlapping width in the first direction to the width of the second sub-image in the first direction is 1:10.

[0240] In some embodiments, the target neural network model is trained by the following steps performed by the processing module:

[0241] Obtain a first training data set and a second training data set; the second training data set is obtained based on the first training data set after screening;

[0242] Training the teacher machine learning model to be trained according to the first training data set to obtain a preliminarily trained teacher machine learning model;

[0243] Training the initially trained teacher machine learning model according to the second training data set to obtain a trained teacher machine learning model;

[0244] According to the second training data set and the trained teacher machine learning model, the knowledge distillation training method is adopted to train the student machine learning model to be trained, and the trained student machine learning model is obtained as the target neural network model.

[0245] In some embodiments, the second training dataset includes a plurality of training images annotated with sample labels;

[0246] The processing module trains the student machine learning model to be trained using a knowledge distillation training method based on the second training data set and the trained teacher machine learning model, and obtains the trained student machine learning model as the target neural network model, including:

[0247] Inputting the training image into the trained teacher machine learning model to obtain a first output result of the trained teacher machine learning model;

[0248] Inputting the training image into the student machine learning model to be trained to obtain a second output result of the student machine learning model to be trained;

[0249] Determining a first loss function according to the first output result and the second output result;

[0250] Determining a second loss function based on the second output result and the sample label of the training image;

[0251] According to the first loss function and the second loss function, a weighted loss function is obtained;

[0252] The parameters of the student machine learning model to be trained are adjusted according to the weighted loss function until the weighted loss function converges, and the trained student machine learning model is obtained as the target neural network model.

[0253] In some embodiments, the first training data set is determined by the following steps performed by the processing module:

[0254] Obtaining an original data set; the original data set includes multiple initial sample images;

[0255] Identify the target object in the initial sample image, and if the target object exists, determine a first reference frame containing the target object;

[0256] updating the position of the first reference frame according to the position information of the first reference frame to obtain a second reference frame;

[0257] determining an overlap degree between the second reference frame and the first reference frame based on the position information of the second reference frame and the position information of the first reference frame;

[0258] According to the comparison result between the overlap degree and the first preset threshold, a sample label is marked for a portion of the sample image of the initial sample image located in the second reference frame, and the portion of the sample image marked with the sample label is used as a training image in the first training data set.

[0259] In some embodiments, the processing module updates the position of the first reference frame according to the position information of the first reference frame to obtain the second reference frame, including:

[0260] According to the position information of the first reference frame, a specific coordinate point in the first reference frame is moved to obtain a third reference frame;

[0261] According to the position information of the third reference frame, the width and height of the third reference frame are adjusted according to the preset scaling factor with the center point of the third reference frame as the center to obtain a fourth reference frame;

[0262] According to the position information of the fourth reference frame, a new center point is determined according to the first preset center point range; and according to the new center point and the preset cropping range, the width and height of the fourth reference frame are adjusted to obtain the second reference frame.

[0263] In some embodiments, the processing module marks a sample label for a portion of the initial sample image located in the second reference frame according to a comparison result between the overlap degree and a first preset threshold, including:

[0264] When the overlap is greater than or equal to the first preset threshold, a first preset range is generated according to the first preset probability, a second preset range is generated according to the second preset probability, a third preset range is generated according to the third preset probability, and a fourth preset range is generated according to the fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; and the sum of the first preset probability, the second preset probability, the third preset probability, and the fourth preset probability is 1;

[0265] When the overlap degree is within a first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with overlap degrees within the first preset range and the total number of training images with positive sample labels in the first training data set is a first preset probability;

[0266] When the overlap degree is within a second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with the overlap degree within the second preset range and the total number of training images with positive sample labels in the first training data set is a second preset probability;

[0267] When the overlap is within a third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with overlap within the third preset range and the total number of training images with positive sample labels in the first training data set is a third preset probability;

[0268] When the overlap degree is within a fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with the overlap degree within the fourth preset range and the total number of training images with positive sample labels in the first training data set is a fourth preset probability;

[0269] The overlap within the first preset range is smaller than the overlap within the second preset range, the overlap within the second preset range is smaller than the overlap within the third preset range, and the overlap within the third preset range is smaller than the overlap within the fourth preset range.

[0270] In some embodiments, in the step of obtaining the first training data set by the processing module, if it is determined that the target object does not exist by performing target object recognition on the initial sample image, the step further includes:

[0271] The initial sample image is labeled with a negative sample label, and the initial sample image labeled with the negative sample label is used as a training image in the first training data set.

[0272] In some embodiments, after the processing module identifies the target object on the initial sample image to determine the presence of the target object and determines a first reference frame containing the target object, the processing module further includes:

[0273] Determining a target center point in the initial sample image according to the size of the initial sample image and in accordance with the second preset center point range, and determining a fifth reference frame with the target center point as the center;

[0274] determining, based on the position information of the fifth reference frame and the position information of the first reference frame, a degree of overlap between the fifth reference frame and any first reference frame in the initial sample image;

[0275] If the degree of overlap between the fifth reference frame and any first reference frame in the initial sample image is less than or equal to a second preset threshold, the portion of the sample image of the initial sample image located in the fifth reference frame is marked as a negative sample label, and the portion of the sample image marked with the negative sample label is used as a training image in the first training data set.

[0276] In some embodiments, the processing module obtains the image to be detected, including:

[0277] Get multiple continuous video frames captured by the camera;

[0278] Based on multiple consecutive video frames, the frame difference method is used to determine whether there is a moving target at the shooting scene of the shooting device. If there is a moving target, the video frame captured by the shooting device is used as the image to be detected.

[0279] In addition, the embodiment of the present disclosure also provides a storage and computing integrated system corresponding to the control method of the monitoring system of the first aspect. Since the principle of solving the problem by the storage and computing integrated system in the embodiment of the present disclosure is similar to the control method of the monitoring system of the first aspect of the embodiment of the present disclosure, the implementation of the storage and computing integrated system can refer to the implementation of the corresponding method of the first aspect, and the repeated parts will not be repeated.

[0280] FIG7 is a schematic diagram of a storage-computing integrated system provided by an embodiment of the present disclosure. As shown in FIG7 , the storage-computing integrated system includes a processing module 71, a storage-computing integrated module 72 and an external storage module 73. Among them, the processing module is, for example, the main processor CPU in the storage-computing integrated system, the storage-computing integrated module can be, for example, a storage-computing core integrated with Flash or SRAM, and the external storage module can be, for example, an external storage module DRAM, which has a faster response speed and reduces the power consumption of the monitoring system. The storage-computing integrated system can be part of the monitoring system. At the same time, the present disclosure designs a lightweight target neural network model, migrates it to the storage-computing integrated module, and adopts a low-power storage-computing integrated method to assist the target neural network in reasoning, complete the wake-up service of the intelligent terminal, reduce power consumption, and have a fast response speed.

[0281] The processing module 71 is configured to obtain an image to be detected; divide the image to be detected into multiple target sub-images according to the shooting angle of the shooting device that shoots the image to be detected, and store them in the external storage module 73;

[0282] The storage and computing module 72 is configured to process each target sub-image among the multiple target sub-images using a pre-trained target neural network model to obtain a recognition result of the target sub-image, and store the recognition result in the external storage module 73;

[0283] The processing module 71 obtains a detection result of whether a target object exists in the image to be detected based on the recognition results corresponding to each target sub-image, and sends the detection result to the terminal so that the terminal determines a display state based on the detection result.

[0284] In some embodiments, the processing module 71 is specifically configured to divide the image to be detected into a first sub-area, a second sub-area and a third sub-area arranged in sequence along the first direction when the shooting angle of view is within a preset angle of view, and the first sub-area and the third sub-area partially overlap with the second sub-area; wherein the width of the first sub-area in the first direction is equal to the width of the third sub-area in the first direction, and the width of the second sub-area in the first direction is greater than the width of the first sub-area in the first direction; the first direction is the height direction of the image to be detected; the part of the image to be detected located in the first sub-area is divided into a plurality of first sub-images arranged side by side along the second direction, and at least part of the first sub-images are in the first direction. The widths in the second direction are equal; the second direction is the width direction of the image to be detected; the portion of the image to be detected located in the second sub-area is divided into a plurality of second sub-images arranged side by side along the second direction, and at least some of the second sub-images have the same widths in the second direction; the portion of the image to be detected located in the third sub-area is divided into a plurality of third sub-images arranged side by side along the second direction, and at least some of the third sub-images have the same widths in the second direction; the target sub-image includes a first sub-image, a second sub-image and a third sub-image; the width of the third sub-image in the second direction is greater than the width of the second sub-image in the second direction, and the width of the second sub-image in the second direction is greater than the width of the first sub-image in the second direction.

[0285] In some embodiments, the first sub-images at least partially overlap in the second direction; the second sub-images at least partially overlap in the second direction; and the third sub-images have no overlap in the second direction.

[0286] In some embodiments, the ratio of the width of the second sub-area to the width of the first sub-area in the first direction is 2:1; the ratio of the width of the third sub-image to the width of the second sub-image in the second direction is 3:2, and the ratio of the width of the second sub-image to the width of the first sub-image in the second direction is 4:3.

[0287] In some embodiments, among the multiple first sub-images arranged side by side along the second direction, except for the first first sub-image and the last first sub-image arranged side by side along the second direction, the widths of the remaining first sub-images in the second direction are the same; the widths of the first first sub-image and the last first sub-image in the second direction are the same, and are smaller than the widths of the remaining first sub-images in the second direction; the ratio of the overlapping width of two adjacent first sub-images in the second direction to the widths of the remaining first sub-images in the second direction is 1:10; among the multiple second sub-images arranged side by side along the second direction, except for the first second sub-image and the last second sub-image arranged side by side along the second direction, the widths of the remaining second sub-images in the second direction are the same; the widths of the first second sub-image and the last second sub-image in the second direction are the same, and are smaller than the widths of the remaining second sub-images in the second direction; the overlapping widths of two adjacent second sub-images in the second direction are The ratio of the overlapping width of two adjacent third sub-images in the second direction to the width of the remaining second sub-images in the second direction is 1:10; among the multiple third sub-images arranged side by side along the second direction, except for the first third sub-image and the last third sub-image arranged side by side along the second direction, the widths of the remaining third sub-images in the second direction are the same; the widths of the first third sub-image and the last third sub-image in the second direction are the same, and are smaller than the widths of the remaining third sub-images in the second direction; the ratio of the overlapping width of two adjacent third sub-images in the second direction to the widths of the remaining third sub-images in the second direction is 1:10; for the first sub-image and the second sub-image adjacent in the first direction, the ratio of the overlapping width in the first direction to the width of the second sub-image in the first direction is 1:10. For the third sub-image and the second sub-image adjacent in the first direction, the ratio of the overlapping width in the first direction to the width of the second sub-image in the first direction is 1:10.

[0288] In some embodiments, the processing module 71 is also configured to train the target neural network model, specifically: obtaining a first training data set and a second training data set; the second training data set is obtained after screening based on the first training data set; training the teacher machine learning model to be trained according to the first training data set to obtain a preliminarily trained teacher machine learning model; training the preliminarily trained teacher machine learning model according to the second training data set to obtain a trained teacher machine learning model; based on the second training data set and the trained teacher machine learning model, using the knowledge distillation training method to train the student machine learning model to be trained, and obtaining the trained student machine learning model as the target neural network model.

[0289] In some embodiments, the second training data set includes multiple training images with labeled sample labels; the processing module 71 is specifically configured to input the training image into the trained teacher machine learning model to obtain a first output result of the trained teacher machine learning model; input the training image into the student machine learning model to be trained to obtain a second output result of the student machine learning model to be trained; determine a first loss function based on the first output result and the second output result; determine a second loss function based on the second output result and the sample label of the training image; obtain a weighted loss function based on the first loss function and the second loss function; adjust the parameters of the student machine learning model to be trained according to the weighted loss function until the weighted loss function converges, and obtain the trained student machine learning model as the target neural network model.

[0290] In some embodiments, the processing module 71 is further configured to determine a first training data set, specifically, to obtain an original data set; the original data set includes multiple initial sample images; the target object is identified in the initial sample image, and if the target object exists, a first reference frame containing the target object is determined; based on the position information of the first reference frame, the position of the first reference frame is updated to obtain a second reference frame; based on the position information of the second reference frame and the position information of the first reference frame, the degree of overlap between the second reference frame and the first reference frame is determined; based on the comparison result between the degree of overlap and the first preset threshold, a sample label is marked for the portion of the sample image of the initial sample image located in the second reference frame, and the portion of the sample image marked with the sample label is used as the training image in the first training data set.

[0291] In some embodiments, in combination with the above embodiments, the processing module 71 is specifically configured to move a specific coordinate point in the first reference frame according to the position information of the first reference frame to obtain a third reference frame; according to the position information of the third reference frame, with the center point of the third reference frame as the center, adjust the width and height of the third reference frame according to a preset scaling factor to obtain a fourth reference frame; according to the position information of the fourth reference frame, determine a new center point according to the first preset center point range; and adjust the width and height of the fourth reference frame according to the new center point and the preset cropping range to obtain a second reference frame.

[0292] In some embodiments, in combination with the above embodiments, the processing module 71 is specifically configured to generate a first preset range according to a first preset probability, generate a second preset range according to a second preset probability, generate a third preset range according to a third preset probability, and generate a fourth preset range according to a fourth preset probability when the overlap degree is greater than or equal to a first preset threshold; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1; when the overlap degree is within the first preset range, determine and mark the part of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with overlap degrees within the first preset range and the total number of training images with positive sample labels in the first training data set is the first preset probability; when the overlap degree is within the second preset range, determine and mark the part of the sample images in the initial sample image located in the second reference frame as positive samples Label, so that the ratio between the number of training images with overlapping degrees within the second preset range and the total number of training images with positive sample labels in the first training data set is a second preset probability; when the overlapping degrees are within the third preset range, determine and mark part of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with overlapping degrees within the third preset range and the total number of training images with positive sample labels in the first training data set is a third preset probability; when the overlapping degrees are within the fourth preset range, determine and mark part of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with overlapping degrees within the fourth preset range and the total number of training images with positive sample labels in the first training data set is a fourth preset probability; the overlapping degrees within the first preset range are smaller than the overlapping degrees within the second preset range, the overlapping degrees within the second preset range are smaller than the overlapping degrees within the third preset range, and the overlapping degrees within the third preset range are smaller than the overlapping degrees within the fourth preset range.

[0293] In some embodiments, the processing module 71 is further configured to, when the target object is identified in the initial sample image and it is determined that the target object does not exist, mark the initial sample image with a negative sample label, and use the initial sample image with the negative sample label as a training image in the first training data set.

[0294] In some embodiments, after identifying the target object in the initial sample image to determine the presence of the target object and determining the first reference frame containing the target object, the processing module 71 is further configured to determine a target center point in the initial sample image according to the size of the initial sample image and the second preset center point range, and determine a fifth reference frame with the target center point as the center; determine the degree of overlap between the fifth reference frame and any first reference frame in the initial sample image according to the position information of the fifth reference frame and the position information of the first reference frame; if the degree of overlap between the fifth reference frame and any first reference frame in the initial sample image is less than or equal to the second preset threshold, the portion of the sample image of the initial sample image located in the fifth reference frame is marked as a negative sample label, and the portion of the sample image marked with the negative sample label is used as a training image in the first training data set.

[0295] In some embodiments, the processing module 71 is specifically configured to obtain multiple continuous video frames captured by the shooting device; based on the multiple continuous video frames, the frame difference method is used to determine whether there is a moving target in the shooting scene of the shooting device. If there is a moving target, the video frame captured by the shooting device is used as the image to be detected.

[0296] In addition, the embodiment of the present disclosure also provides a control device for the monitoring system corresponding to the control method of the monitoring system of the second aspect. Since the principle of solving the problem by the control device of the monitoring system in the embodiment of the present disclosure is similar to the control method of the monitoring system of the second aspect of the embodiment of the present disclosure, the implementation of the control device of the monitoring system can refer to the implementation of the corresponding method of the second aspect, and the repeated parts will not be repeated.

[0297] FIG8 is a schematic diagram of the control device of the monitoring system provided by an embodiment of the present disclosure. As shown in FIG8 , the storage-computing integrated system 81 and the terminal 82; the storage-computing integrated system includes a processing module 811, a storage-computing integrated module 812 and an external storage module 813. Among them, the processing module is, for example, the main processor CPU in the storage-computing integrated system, the storage-computing integrated module can be, for example, a storage-computing core integrated with Flash or SRAM, and the external storage module can be, for example, an external storage module DRAM, which has a faster response speed and reduces the power consumption of the monitoring system. The storage-computing integrated system can be part of the monitoring system. At the same time, the present disclosure designs a lightweight target neural network model, migrates it to the storage-computing integrated module, and adopts a low-power storage-computing integrated method to assist the target neural network in reasoning, complete the wake-up service of the intelligent terminal, reduce power consumption, and have a fast response speed.

[0298] The processing module 811 is configured to obtain an image to be detected; divide the image to be detected into multiple target sub-images according to the shooting angle of the shooting device that shoots the image to be detected, and store them in the external storage module 813.

[0299] The storage and computing module 812 is configured to read the target sub-image; use the pre-trained target neural network model to process the target sub-image, obtain the recognition result of the target sub-image, and store the recognition result in the external storage module 813.

[0300] The processing module 811 is configured to obtain a detection result of whether a target object exists in the image to be detected based on the recognition results corresponding to each target sub-image, and send the detection result to the terminal 82 .

[0301] The terminal 82 is configured to determine a display state based on the detection result.

[0302] In some embodiments, the terminal 82 includes a display module 821 and a main control module 822; the main control module 822 is configured to send a wake-up request to the display module and reset the sleep timer when it is determined that the detection result indicates that there is a target object in the image to be detected, in response to the terminal request to receive the detection result sent by the processing module; when it is determined that the detection result indicates that there is no target object in the image to be detected, and the current system time is greater than the preset time from the last time the display module was awakened, send a sleep request to the display module and control itself to enter a sleep state; the display module 821 is configured to respond to the wake-up request and determine that the display state is to normally display the shooting picture of the shooting device; and respond to the sleep request and determine that the display state is to sleep.

[0303] In some embodiments, the control device of the monitoring system further includes a shooting device 83 , and the image segmentation module 811 can obtain data from the low-power shooting device 83 through protocols such as SPI.

[0304] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium is further provided. The non-transitory computer-readable storage medium stores a computer program, wherein when the program is executed by a processor, the program implements the steps of the control method of the monitoring system in any of the above embodiments.

[0305] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the system of the present disclosure are executed.

[0306] It should be noted that the computer non-transitory readable medium shown in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any non-transitory computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the non-transitory computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0307] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the aforementioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two connected boxes can actually represent execution in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0308] It is understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Those skilled in the art may make various modifications and improvements without departing from the spirit and substance of the present disclosure, and such modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A control method for a monitoring system, wherein: include: Acquire the image to be detected; Dividing the image to be detected into a plurality of target sub-images according to the shooting angle of view of a shooting device that shoots the image to be detected; For each of the multiple target sub-images, a pre-trained target neural network model is used to process the target sub-image to obtain a recognition result of the target sub-image; Based on the recognition results corresponding to the target sub-images, a detection result of whether a target object exists in the image to be detected is obtained, and the detection result is sent to a terminal, so that the terminal determines a display state at least based on the detection result.

2. The control method of the monitoring system according to claim 1, wherein: The step of dividing the image to be detected into a plurality of target sub-images according to the shooting angle of the shooting device comprises: When the shooting angle of view is within a preset angle of view, the image to be detected is divided into a first sub-region, a second sub-region and a third sub-region sequentially arranged along a first direction, and the first sub-region and the third sub-region partially overlap with the second sub-region; wherein the width of the first sub-region in the first direction is equal to the width of the third sub-region in the first direction, and the width of the second sub-region in the first direction is greater than the width of the first sub-region in the first direction; and the first direction is a height direction of the image to be detected; Divide the portion of the image to be detected located in the first sub-area into a plurality of first sub-images arranged side by side along a second direction, wherein at least some of the first sub-images have equal widths in the second direction; the second direction is the width direction of the image to be detected; Dividing the portion of the image to be detected located in the second sub-area into a plurality of second sub-images arranged side by side along the second direction, at least some of the second sub-images having equal widths in the second direction; Divide the portion of the image to be detected located in the third sub-area into a plurality of third sub-images arranged side by side along the second direction, at least some of the third sub-images having equal widths in the second direction; the target sub-image includes a first sub-image, a second sub-image and a third sub-image; The width of the third sub-image in the second direction is greater than the width of the second sub-image in the second direction, and the width of the second sub-image in the second direction is greater than the width of the first sub-image in the second direction.

3. The control method of the monitoring system according to claim 2, wherein: The first sub-images at least partially overlap in the second direction; the second sub-images at least partially overlap in the second direction; The third sub-images have no overlap in the second direction.

4. The control method of the monitoring system according to claim 3, wherein: The ratio of the width of the second sub-region to the width of the first sub-region in the first direction is 2:1; The ratio of the width of the third sub-image to that of the second sub-image in the second direction is 3:2, and the ratio of the width of the second sub-image to that of the first sub-image in the second direction is 4:

3.

5. The control method of the monitoring system according to claim 3, wherein: Among the multiple first sub-images arranged side by side along the second direction, except for the first first sub-image and the last first sub-image arranged side by side along the second direction, the widths of the other first sub-images in the second direction are the same; the widths of the first first sub-image and the last first sub-image in the second direction are the same and smaller than the widths of the other first sub-images in the second direction; the ratio of the overlapping width of two adjacent first sub-images in the second direction to the widths of the other first sub-images in the second direction is 1:10; Among the plurality of second sub-images arranged side by side along the second direction, except for the first second sub-image and the last second sub-image arranged side by side along the second direction, the widths of the remaining second sub-images in the second direction are the same; the widths of the first second sub-image and the last second sub-image in the second direction are the same and smaller than the widths of the remaining second sub-images in the second direction; the ratio of the overlapping width of two adjacent second sub-images in the second direction to the widths of the remaining second sub-images in the second direction is 1:10; Among the plurality of third sub-images arranged side by side along the second direction, except for the first third sub-image and the last third sub-image arranged side by side along the second direction, the widths of the remaining third sub-images in the second direction are the same; the widths of the first third sub-image and the last third sub-image in the second direction are the same and smaller than the widths of the remaining third sub-images in the second direction; the ratio of the overlapping width of two adjacent third sub-images in the second direction to the widths of the remaining third sub-images in the second direction is 1:10; The first sub-image and the second sub-image adjacent in the first direction have an overlapping width in the first direction and a width of the second sub-image in the first direction in a ratio of 1:10; the third sub-image and the second sub-image adjacent in the first direction have an overlapping width in the first direction and a width of the second sub-image in the first direction in a ratio of 1:

10.

6. The control method of the monitoring system according to claim 1, wherein: The target neural network model is trained by the following steps: Acquire a first training data set and a second training data set; the second training data set is obtained after screening the first training data set; Training the teacher machine learning model to be trained according to the first training data set to obtain a preliminarily trained teacher machine learning model; Training the preliminarily trained teacher machine learning model according to the second training data set to obtain a trained teacher machine learning model; According to the second training data set and the trained teacher machine learning model, the student machine learning model to be trained is trained using a knowledge distillation training method to obtain the trained student machine learning model as the target neural network model.

7. The control method of the monitoring system according to claim 6, wherein: The second training data set includes a plurality of training images annotated with sample labels; The method of training the student machine learning model to be trained by using the knowledge distillation training method according to the second training data set and the trained teacher machine learning model to obtain the trained student machine learning model as the target neural network model includes: Inputting the training image into the trained teacher machine learning model to obtain a first output result of the trained teacher machine learning model; Inputting the training image into the student machine learning model to be trained to obtain a second output result of the student machine learning model to be trained; Determine a first loss function according to the first output result and the second output result; Determining a second loss function according to the second output result and the sample label of the training image; Obtaining a weighted loss function according to the first loss function and the second loss function; The parameters of the student machine learning model to be trained are adjusted according to the weighted loss function until the weighted loss function converges, thereby obtaining the trained student machine learning model as the target neural network model.

8. The control method of the monitoring system according to claim 7, wherein: The first training data set is determined by the following steps: Acquire an original data set; the original data set includes a plurality of initial sample images; Identify the target object on the initial sample image, and if the target object exists, determine a first reference frame containing the target object; According to the position information of the first reference frame, the position of the first reference frame is updated to obtain a second reference frame; determining an overlap degree between the second reference frame and the first reference frame according to the position information of the second reference frame and the position information of the first reference frame; According to the comparison result between the overlap degree and the first preset threshold, the sample label is marked for the partial sample image of the initial sample image located in the second reference frame, and the partial sample image marked with the sample label is used as the training image in the first training data set.

9. The control method of the monitoring system according to claim 8, wherein: The updating the position of the first reference frame according to the position information of the first reference frame to obtain the second reference frame includes: According to the position information of the first reference frame, move the specific coordinate point in the first reference frame to obtain a third reference frame; According to the position information of the third reference frame, taking the center point of the third reference frame as the center and according to a preset scaling factor, the width and height of the third reference frame are adjusted to obtain a fourth reference frame; According to the position information of the fourth reference frame, a new center point is determined according to the first preset center point range; and according to the new center point and the preset cropping range, the width and height of the fourth reference frame are adjusted to obtain the second reference frame.

10. The control method of the monitoring system according to claim 8, wherein: The step of marking the sample label for a portion of the sample image of the initial sample image located in the second reference frame according to a comparison result between the overlap degree and a first preset threshold value includes: When the overlap is greater than or equal to the first preset threshold, a first preset range is generated according to a first preset probability, a second preset range is generated according to a second preset probability, a third preset range is generated according to a third preset probability, and a fourth preset range is generated according to a fourth preset probability; the first preset probability is greater than the second preset probability, the second preset probability is greater than the third preset probability, and the third preset probability is greater than or equal to the fourth preset probability; the sum of the first preset probability, the second preset probability, the third preset probability and the fourth preset probability is 1; When the overlap degree is within the first preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with the overlap degree within the first preset range and the total number of training images with the positive sample labels in the first training data set is the first preset probability; When the overlap degree is within the second preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with the overlap degree within the second preset range and the total number of training images with the positive sample labels in the first training data set is the second preset probability; When the overlap degree is within the third preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with the overlap degree within the third preset range and the total number of training images with the positive sample labels in the first training data set is the third preset probability; When the overlap degree is within the fourth preset range, determining and marking a portion of the sample images in the initial sample image located in the second reference frame as positive sample labels, so that the ratio between the number of training images with the overlap degree within the fourth preset range and the total number of training images with the positive sample labels in the first training data set is the fourth preset probability; The overlap within the first preset range is smaller than the overlap within the second preset range, the overlap within the second preset range is smaller than the overlap within the third preset range, and the overlap within the third preset range is smaller than the overlap within the fourth preset range.

11. The control method of the monitoring system according to claim 8, wherein: In the step of obtaining the first training data set, when the target object is identified on the initial sample image and it is determined that the target object does not exist, the step further includes: The initial sample image is labeled with a negative sample label, and the initial sample image labeled with the negative sample label is used as a training image in the first training data set.

12. The control method of the monitoring system according to claim 8, wherein: After identifying the target object on the initial sample image to determine that the target object exists, and determining a first reference frame containing the target object, the method further includes: According to the size of the initial sample image and in accordance with the second preset center point range, a target center point is determined in the initial sample image, and a fifth reference frame is determined with the target center point as the center; determining, according to the position information of the fifth reference frame and the position information of the first reference frame, a degree of overlap between the fifth reference frame and any first reference frame in the initial sample image; If the degree of overlap between the fifth reference frame and any first reference frame in the initial sample image is less than or equal to a second preset threshold, the portion of the sample image of the initial sample image located in the fifth reference frame is marked as a negative sample label, and the portion of the sample image marked with the negative sample label is used as a training image in the first training data set.

13. The control method of the monitoring system according to claim 1, wherein: The step of acquiring the image to be detected comprises: Get multiple continuous video frames captured by the camera; According to the multiple continuous video frames, a frame difference method is used to determine whether there is a moving target at the shooting scene of the shooting device. If there is a moving target, the video frames shot by the shooting device are used as images to be detected.

14. A control method for a monitoring system, wherein: The monitoring and broadcasting system includes a storage and computing integrated system and a terminal; the storage and computing integrated system includes a processing module, a storage and computing integrated module and an external storage module; the control method of the monitoring and broadcasting system includes: The processing module acquires an image to be detected; divides the image to be detected into a plurality of target sub-images according to the shooting angle of the shooting device that shoots the image to be detected, and stores the sub-images in the external storage module; The storage-computation integrated module reads the target sub-image; uses a pre-trained target neural network model to process the target sub-image to obtain a recognition result of the target sub-image, and stores the recognition result in the external storage module; The processing module obtains a detection result of whether a target object exists in the image to be detected based on the recognition results corresponding to each of the target sub-images, and sends the detection result to the terminal; The terminal determines a display state based on the detection result.

15. The control method of the monitoring system according to claim 14, wherein: The terminal includes a display module and a main control module; The terminal determines a display state based on the detection result, including: When the main control module determines that the detection result indicates that the target object exists in the image to be detected, the main control module sends a wake-up request to the display module and resets the sleep timer to respond to the terminal request to receive the detection result sent by the processing module; when the main control module determines that the detection result indicates that the target object does not exist in the image to be detected, and the time from the current system time to the last time the display module was awakened is greater than a preset time, the main control module sends a sleep request to the display module and controls itself to enter a sleep state; The display module responds to the wake-up request to determine that the display state is to normally display the captured image of the capturing device; and responds to the sleep request to determine that the display state is to sleep.

16. A storage and computing integrated system, comprising a processing module, a storage and computing integrated module and an external storage module; The processing module is configured to obtain an image to be detected; divide the image to be detected into a plurality of target sub-images according to a shooting angle of a shooting device that shoots the image to be detected, and store the sub-images in the external storage module; The storage and computing integrated module is configured to read the target sub-image; Using a pre-trained target neural network model, the target sub-image is processed to obtain a recognition result of the target sub-image, and the recognition result is stored in the external storage module; The processing module is configured to obtain a detection result of whether a target object exists in the image to be detected based on the recognition results corresponding to each of the target sub-images, and send the detection result to the terminal so that the terminal determines a display state based on the detection result.

17. A control device for a monitoring system, comprising a storage and computing system and a terminal; the storage and computing system comprises a processing module, a storage and computing module and an external storage module; The processing module is configured to obtain an image to be detected; divide the image to be detected into a plurality of target sub-images according to a shooting angle of a shooting device that shoots the image to be detected, and store the sub-images in the external storage module; The storage and computing integrated module is configured to read the target sub-image; Using a pre-trained target neural network model, the target sub-image is processed to obtain a recognition result of the target sub-image, and the recognition result is stored in the external storage module; The processing module is configured to obtain a detection result of whether a target object exists in the image to be detected based on the recognition results corresponding to each of the target sub-images, and send the detection result to the terminal; The terminal is configured to determine a display state based on the detection result.

18. A computer non-transitory readable storage medium, wherein: The computer non-volatile readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the control method of the monitoring system as described in any one of claims 1 to 13, and / or the steps of the control method of the monitoring system as described in any one of claims 14 to 15.

Images