Progress Detection Method, Device, Storage Medium and Program Product

Abstract

An embodiment of the present application provides a progress detection method, device, storage medium, and program product. The method includes obtaining a first image and a second image. The first image is obtained by a black-and-white camera at the current moment, and the second image is obtained by an event camera between the current moment and the previous moment. The previous moment is the moment when the black-and-white camera captures a third image adjacent to the first image before the current moment. Preprocess the first image and the second image to obtain the current image frame. Input the current image frame into a deep learning model to obtain the current progress result and the current prediction result corresponding to the current image frame, and push the current progress result and the current prediction result to the target user. The method provided by the embodiment of the present application can very efficiently and accurately master the current progress and predicted progress of the activity without the need to go to the site to check.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of medical devices, and in particular, to a progress detection method, device, storage medium, and program product. Background Art

[0002] For different medical activities, different countermeasures are taken at different progress stages.

[0003] In the related art, the progress of activities is usually detected by manual observation.

[0004] However, in the process of implementing the present application, the inventors found that there are at least the following problems in the prior art: The above method requires on-site inspection, which is time-consuming and laborious, and often makes mistakes. Summary of the Invention

[0005] The embodiments of the present application provide a progress detection method, device, storage medium, and program product to improve efficiency and accuracy.

[0006] In a first aspect, the embodiments of the present application provide a progress detection method, including:

[0007] Obtain a first image and a second image; the first image is obtained by a black-and-white camera at the current moment; the second image is obtained by an event camera between the current moment and the previous moment; the previous moment is the moment when the black-and-white camera captures a third image adjacent to the first image before the current moment;

[0008] Preprocess the first image and the second image to obtain a current image frame;

[0009] Input the current image frame into a deep learning model to obtain a current progress result and a current prediction result corresponding to the current image frame;

[0010] Push the current progress result and the current prediction result to a target user.

[0011] In a possible design, the preprocessing the first image and the second image to obtain a current image frame includes:

[0012] Convert the first image to obtain a first image frame with a preset structure; the preset structure is related to the data input format of the deep learning model;

[0013] Convert the second image to obtain a second image frame with the preset structure;

[0014] Connect the first image frame and the second image frame to obtain the current image frame with the preset structure.

[0015] In a possible design, the conversion of the first image to obtain a first image frame with a preset structure includes:

[0016] Input the first image into a convolutional neural network to obtain a first image frame with a preset structure.

[0017] In a possible design, the conversion of the second image to obtain a second image frame with a preset structure includes:

[0018] Construct a target three-dimensional solid based on the first image and the third image; the first dimension of the target three-dimensional solid is one dimension of the first image and the third image, the second dimension of the target three-dimensional solid is the other dimension of the second image and the third image, and the third dimension of the target three-dimensional solid is the time dimension where the current moment and the previous moment are located;

[0019] Slice the target three-dimensional solid to obtain multiple sub-solids; the dimensional planes of the sub-solids are parallel or coincident with the dimensional planes of the target three-dimensional solid;

[0020] Convert the second image into the target three-dimensional solid, and for each sub-solid, determine the value corresponding to the sub-solid based on the data corresponding to the sub-solid in the second image;

[0021] Based on a preset rule, cascade the values corresponding to the multiple sub-solids to obtain a second image frame.

[0022] In a possible design, the input of the current image frame into a deep learning model to obtain the current progress result and the current prediction result corresponding to the current image frame includes:

[0023] Perform encoding processing, decoding processing, and fully connected processing on the current image frame in sequence to obtain the current progress result;

[0024] Determine the current prediction result based on the current progress result and the previous prediction result.

[0025] In a possible design, the performance of encoding processing, decoding processing, and fully connected processing on the current image frame in sequence to obtain the current progress result includes:

[0026] Perform encoding processing on the current image frame to obtain an encoding result;

[0027] Perform decoding processing on the encoding result and the previous output result to obtain the current output result; the previous output result is the current output result corresponding to the previous image frame;

[0028] Perform fully connected processing on the current output result to obtain the current progress result;

[0029] Determining the current prediction result according to the current progress result and the previous prediction result includes:

[0030] Determining the current prediction deviation according to the current progress result and the previous prediction result;

[0031] Connecting the current output result and the current prediction deviation and then performing a fully connected process to obtain the current prediction result.

[0032] In a possible design, pushing the current progress result and the current prediction result to the target user includes:

[0033] Obtaining the identification information of the detection object corresponding to the first image and the second image;

[0034] Generating an alarm message according to the prediction result;

[0035] Based on the identification information, pushing the current progress result, the current prediction result and the alarm message to the target user.

[0036] In a second aspect, an embodiment of the present application provides a progress detection method device, including:

[0037] An acquisition module, configured to acquire a first image and a second image; the first image is obtained by a black and white camera at the current moment; the second image is obtained by an event camera between the current moment and the previous moment; the previous moment is the moment when the black and white camera captures a third image adjacent to the first image before the current moment;

[0038] A processing module, configured to preprocess the first image and the second image to obtain a current image frame;

[0039] An input module, configured to input the current image frame into a deep learning model to obtain a current progress result and a current prediction result corresponding to the current image frame;

[0040] A pushing module, configured to push the current progress result and the current prediction result to the target user.

[0041] In a third aspect, an embodiment of the present application provides a progress detection method device, including: at least one processor and a memory;

[0042] The memory stores computer execution instructions;

[0043] The at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the methods described in the first aspect and various possible designs of the first aspect above.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in the first aspect above and all possible designs of the first aspect.

[0045] In a fifth aspect, an embodiment of the present application provides a computer program product including a computer program, which, when executed by a processor, implements the method described in the first aspect above and all possible designs of the first aspect.

[0046] The progress detection method, device, storage medium, and program product provided in this embodiment include obtaining a first image and a second image. The first image is obtained by a black-and-white camera at the current moment, and the second image is obtained by an event camera between the current moment and the previous moment. The previous moment is the moment when the black-and-white camera captured a third image adjacent to the first image before the current moment. Preprocess the first image and the second image to obtain the current image frame, input the current image frame into a deep learning model to obtain the current progress result and the current prediction result corresponding to the current image frame, and push the current progress result and the current prediction result to the target user. The method provided in the embodiment of the present application can collect images of the detection object in the activity by using a camera, and after inputting the images into a deep learning model, it can obtain the corresponding current progress result and the current prediction result in real time, and very efficiently and accurately master the current progress and the predicted progress of the activity without having to go to the site to check. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0048] Figure 1 It is a schematic diagram of the application scenario of the progress detection method provided in the embodiment of the present application;

[0049] Figure 2 It is a schematic flowchart of the progress detection method provided in the embodiment of the present application;

[0050] Figure 3 It is a schematic structural diagram of the data acquisition device provided in the embodiment of the present application;

[0051] Figure 4a It is a schematic structural diagram of the target three-dimensional corresponding to the second image provided in the embodiment of the present application;

[0052] Figure 4bSchematic diagram of the principle of the disaggregated cascade processing corresponding to the second image provided by the embodiment of the present application;

[0053] Figure 5 Schematic diagram of the principle of preprocessing the first image and the second image provided by the embodiment of the present application Figure 1 ;

[0054] Figure 6 Schematic diagram of the principle of the deep learning model provided by the embodiment of the present application;

[0055] Figure 7 Schematic diagram of the cell growth stage provided by the embodiment of the present application;

[0056] Figure 8 Schematic diagram of the terminal interface of the user terminal provided by the embodiment of the present application;

[0057] Figure 9 Schematic diagram of the structure of the progress detection method device provided by the embodiment of the present application;

[0058] Figure 10 Schematic diagram of the hardware structure of the progress detection method device provided by the embodiment of the present application. Detailed implementation manners

[0059] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0060] It should be noted that the progress detection method and device provided by the embodiments of the present application can be used in the medical device field, and can also be used in any field other than the medical device field. The application fields of the progress detection method and device provided by the present application are not limited.

[0061] There are situations where devices operate automatically in multiple links during medical processes, such as intravenous infusion, cell culture, hemodialysis, etc. And different response measures need to be taken according to the progress of the activities. For example, during intravenous infusion, if the liquid is about to run out, the infusion device needs to be turned off or the liquid needs to be replaced. During cell culture, if the cells grow to a certain stage, the cells need to be detected. During hemodialysis, when the coagulation rate reaches a certain level, anticoagulants need to be added.

[0062] In the related art, usually medical staff personally arrive at the scene to check and confirm the progress, and give specific coping strategies according to the current progress. However, medical staff have a heavy workload and limited energy, and shuttling between different wards and beds is time-consuming and laborious, and it is very easy to make mistakes.

[0063] To solve the above technical problems, the inventors of the present application found through research that if a monitoring device is used to automatically observe the progress, obtain the progress result and push the progress result to the medical staff, there is no need for the medical staff to go to the scene for real-time viewing, which can greatly reduce the workload of the staff. In addition, by predicting the progress result, it can provide a data basis for the medical staff to reasonably arrange working hours. Further, considering the existence of individual differences, a deep learning model with higher accuracy can be used to process the images of the detected objects collected to detect and predict the results. Based on this, an embodiment of the present application provides a progress detection method.

[0064] Figure 1 It is a schematic diagram of the application scenario of the progress detection method provided by the embodiment of the present application. As Figure 1 shown, the data acquisition terminal and the user terminal are both communicatively connected to the server. Optionally, the user terminal can be a device such as a computer, a tablet computer, or a mobile phone. The data acquisition terminal can include a black and white camera and an event camera. Optionally, considering the stability of the ambient light in the environment where the detected object is located, the data acquisition terminal can also include an infrared light source and a light-shielding darkroom, and the infrared light source, the black and white camera, and the actual camera are arranged in the light-shielding darkroom to obtain stable ambient light, which is beneficial to improving the accuracy of the collected images.

[0065] In the specific implementation process, the black and white camera of the data acquisition terminal acquires the first image at the current moment, and the event camera acquires the second image between the first image and the previous adjacent image frame. The server acquires the first image and the second image, preprocesses the first image and the second image to obtain the current image frame, inputs the current image frame into the deep learning model to obtain the current progress result and the current prediction result corresponding to the current image frame, and pushes the current progress result that conforms to the current prediction result to the user terminal corresponding to the target user. The progress detection method provided by the embodiment of the present application can efficiently and accurately master the current progress and predicted progress of the activity by collecting the images of the detected objects in the activity by using a camera and inputting the images into the deep learning model, without the need to go to the scene for viewing.

[0066] It should be noted that Figure 1The schematic diagram of the scene shown is only an example. The progress detection method and the scene described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those of ordinary skill in the art know that with the evolution of the system and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0067] The following uses specific embodiments to elaborate on the technical solutions of the present application in detail. These several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0068] Figure 2 It is a schematic flowchart of the progress detection method provided by the embodiments of the present application. As Figure 2 shown, the method includes:

[0069] 201. Obtain a first image and a second image; the first image is obtained by a black-and-white camera at the current moment; the second image is obtained by an event camera between the current moment and the previous moment; the previous moment is the moment when the black-and-white camera captured a third image adjacent to the first image before the current moment.

[0070] The execution subject of this embodiment can be a terminal device or a server, such as Figure 1 the server shown.

[0071] Specifically, the black-and-white camera performs real-time continuous shooting on the detection object based on a preset shooting frequency to obtain continuous image frames. The event camera is a bionic sensor with a microsecond response time, which can record the asynchronous stream of the brightness change of each pixel, called an "event". The event camera generates an event by detecting the brightness change of each pixel. Compared with traditional cameras, it is more suitable for use in high-dynamic and high-speed environments, and has the advantages of high dynamic range (High Dynamic Range Imaging, HDR), high time resolution, and no motion blur. By using a black-and-white camera, it is possible to ensure a full-view observation of the status of the target being photographed.

[0072] In some embodiments, in order to provide stable ambient light for image shooting to ensure the accuracy of data collection, as Figure 3As shown, the data acquisition device may further include a light-shielding darkroom and an infrared light source. Among them, the light-shielding darkroom provides the container space where the entire imaging system is located and does not allow external light to enter. Some targets that are sensitive to changes in external ambient light need to be processed in a light-shielding darkroom. For insensitive situations, the target area can also be photographed without being affected by adjusting the camera's FOV and focal length. In order to cooperate with the event camera, the infrared light source can be a constantly-on infrared light source (non-PWM dimming), with a wavelength between 850 nm and 960 nm, and irradiating the object to be photographed directly. The event camera is on the opposite side of the infrared light source and receives the infrared light that has passed through the object to be photographed. Two black-and-white cameras can be set, respectively on both sides of the infrared light source, to ensure that the target situation to be photographed can be seen from all perspectives.

[0073] 202. Preprocess the first image and the second image to obtain the current image frame.

[0074] Specifically, after obtaining the first image and the second image, the first image and the second image can be fused. The image obtained by the event camera during the time period corresponding to adjacent image frames captured by the black-and-white camera is used as the second image, and the first image at the current moment and the second image are fused to obtain the fused current image frame. Based on this, the current image frame includes both the content captured by the black-and-white camera and the content captured by the event camera.

[0075] In some embodiments, preprocessing the first image and the second image to obtain the current image frame may include: converting the first image to obtain a first image frame with a preset structure; the preset structure is related to the data input format of the deep learning model; converting the second image to obtain a second image frame with the preset structure; connecting the first image frame and the second image frame to obtain the current image frame with the preset structure.

[0076] Specifically, the first image and the second image can be respectively subjected to conversion processing to obtain a first image frame and a second image frame with a preset structure. Furthermore, the first image frame and the second image frame can be connected end to end to obtain the current image frame. During the end-to-end connection process, the order of the first image frame and the second image frame can be interchanged, and this embodiment does not limit this.

[0077] In some embodiments, converting the first image to obtain a first image frame with a preset structure may include: inputting the first image into a convolutional neural network to obtain a first image frame with a preset structure.

[0078] Specifically, assume that the preset result is 1*1*n, and the data dimension of the first image is H*W*1, where H and W are integers greater than 1. To convert the data structure of the first image, the first image can be input into a convolutional neural network for convolutional processing of the first image to obtain a data structure of the first image frame as 1*1*n. The convolutional neural network includes multiple convolutional layers and pooling modules, etc.

[0079] In some embodiments, converting the second image to obtain a second image frame with a preset structure may include: constructing a target three-dimensional solid based on the first image and the third image; a first dimension of the target three-dimensional solid is one dimension of the first image and the third image, a second dimension of the target three-dimensional solid is the other dimension of the second image and the third image, and a third dimension of the target three-dimensional solid is the time dimension where the current moment and the previous moment are located; slicing the target three-dimensional solid to obtain a plurality of sub-solids; a dimension plane of the sub-solid is parallel or coincident with a dimension plane of the target three-dimensional solid; converting the second image into the target three-dimensional solid, and for each sub-solid, determining a value corresponding to the sub-solid based on data corresponding to the sub-solid obtained in the second image; based on a preset rule, cascading the values corresponding to the plurality of sub-solids to obtain a second image frame.

[0080] Specifically, the second image is a segment between the first image at the current moment and the third image at the previous moment. As Figure 4a shown, the data of the second image is included between the first image (plane filled with diagonal lines) collected at the current moment t1 and the third image (plane filled with dots) collected at the previous moment t0. To convert the second image into a preset structure of 1*1*n, as Figure 4b shown, the three-dimensional solid constructed from the first image and the third image can be broken into several sub-blocks based on the spatio-temporal relationship, and the difference between the positive signal and the negative signal of each sub-solid is taken as the result of the sub-solid, and the results of each sub-solid are cascaded together as the representation of the second image.

[0081] Exemplarily, as Figure 5 shown, the first image image obtained by a black-and-white camera can be processed by a convolutional neural network CNNNet to obtain a first image frame, and the second image event obtained by an event camera is processed by breaking and cascading the representation to obtain a second image frame. The first image frame and the second image frame are connected to obtain the input data of the deep learning model, the current image frame.

[0082] 203. Input the current image frame into the deep learning model to obtain the current progress result and the current prediction result corresponding to the current image frame.

[0083] Specifically, after the current image frame that meets the data format requirements, the current image frame can be input into the deep learning model, and the deep learning model processes the current image frame to obtain the current progress result corresponding to the current image frame, and the current prediction result can be predicted.

[0084] In this embodiment, since the input of the model is multi-modal information (including image data obtained by a black-and-white camera and an event camera respectively) and includes temporal prediction, a network architecture based on transform can be adopted.

[0085] In some embodiments, inputting the current image frame into the deep learning model to obtain the current progress result and the current prediction result corresponding to the current image frame may include: sequentially performing encoding processing, decoding processing, and fully connected processing on the current image frame to obtain the current progress result; determining the current prediction result according to the current progress result and the previous prediction result.

[0086] In some embodiments, sequentially performing encoding processing, decoding processing, and fully connected processing on the current image frame to obtain the current progress result may include: performing encoding processing on the current image frame to obtain an encoding result; performing decoding processing on the encoding result and the previous output result to obtain the current output result; the previous output result is the current output result corresponding to the previous image frame; performing fully connected processing on the current output result to obtain the current progress result; determining the current prediction result according to the current progress result and the previous prediction result may include: determining the current prediction deviation according to the current progress result and the previous prediction result; connecting the current output result and the current prediction deviation and then performing fully connected processing to obtain the current prediction result.

[0087] Exemplarily, as Figure 6 shown, the deep learning model includes an encoder Encoder, a decoder Decoder, a first fully connected module Linear1 (Linear + activation function), a concatenation module concatenate, and a second fully connected module Linear2. In the specific implementation process, the input signal is sent to the Encoder for encoding, and the previous output result lastoutput and the encoding result of the Encoder are jointly input into the Decoder for processing to obtain the current output result output. The output passes through Linear1 to obtain the current progress result. The previous prediction result is compared with the current progress result to obtain the current prediction deviation, which is added to the deviation sequence. The current output result output and the prediction deviation are connected through concatenate and then pass through Linear2 to obtain the current prediction result.

[0088] In some embodiments, in the process of training a pre-trained model to obtain a deep learning model, it is first necessary to prepare a sample set. Specifically, a large number of sequences of target images (such as petri dishes in the case of cell culture, etc.) can be collected. Combining the imaging situation at the initial moment and the real-time imaging during the process, the area and color change degree of the features of interest in each frame of the image are judged, and the calculation index corresponding to the current frame is output. Considering that due to individual differences, there are certain gaps in information such as the depth of the color corresponding to the target, so there may be certain inaccuracies in directly using an algorithm for image binarization in this process. Therefore, manual annotation needs to be combined. For each corresponding image sequence, it is not necessary to perform real-time annotation on all of them, and only a few key frames of the image need to be selected to calibrate the key parameters.

[0089] Exemplarily, as Figure 7 shown, taking the cell and the cell growth situation in the culture as an example, the progress can be divided into stages, such as the initial stage, the initial slow growth stage, and the large growth stage. Further, it is only necessary to annotate the key image frames of different stages such as the initial stage, the initial slow growth stage, and the large growth stage.

[0090] 204. Push the current progress result and the current prediction result to the target user.

[0091] Specifically, after the data acquisition device acquires the image, it can be packaged and sent to the server through the Internet of Things. The method is implemented by the server in the cloud. After obtaining the current progress result and the current prediction result, the results can be sent to the terminal device of the corresponding medical staff (or experimenter) through the Internet of Things.

[0092] In some embodiments, pushing the current progress result and the current prediction result to the target user may include: obtaining the identification information of the detection object corresponding to the first image and the second image; generating an alarm message according to the current prediction result; based on the identification information, pushing the current progress result, the current prediction result, and the alarm message to the target user.

[0093] Specifically, in order to remind the user and facilitate the user to make a timely response, an alarm message can be generated based on the current prediction result. Specifically, an alarm threshold and a warning threshold can be set. After the current index value is higher than the alarm threshold, the corresponding information becomes red, and according to the terminal settings, a continuous sound or vibration warning is issued until the medical staff processes it and closes the alarm. Exemplarily, as Figure 8 shown, the terminal display interface may include bed information, start time, and current and predicted index information. And for the information higher than the warning threshold, the corresponding information can be specially displayed. For example, it can be emphasized or displayed in a specific color, and intermittent vibration or sound alarms can also be carried out at the same time until the medical staff processes it.

[0094] During the treatment process by medical staff, the corresponding treatment methods (such as increasing the dosage of anticoagulants, etc.) can be recorded in the system. After the end of the current treatment (or experiment), it is possible to judge when and what corresponding treatment should be done next based on the time dosage and time curve of the current treatment.

[0095] The progress detection method provided in this embodiment can collect images of the detection object during the activity by using a camera, and after inputting the images into the deep learning model, it can obtain the corresponding current progress result and current prediction result in real time, and very efficiently and accurately master the current progress and predicted progress of the activity without having to go to the site to check.

[0096] Figure 9 It is a schematic structural diagram of the progress detection method device provided in the embodiment of the present application. As Figure 9 shown, the progress detection method device 90 includes: an acquisition module 901, a processing module 902, an input module 903, and a push module 904.

[0097] The acquisition module 901 is used to acquire a first image and a second image; the first image is obtained by a black-and-white camera shooting at the current moment; the second image is obtained by an event camera shooting between the current moment and the previous moment; the previous moment is the moment when the black-and-white camera shoots a third image adjacent to the first image before the current moment.

[0098] The processing module 902 is used to preprocess the first image and the second image to obtain a current image frame.

[0099] The input module 903 is used to input the current image frame into the deep learning model to obtain the current progress result and current prediction result corresponding to the current image frame.

[0100] The push module 904 is used to push the current progress result and the current prediction result to the target user.

[0101] The progress detection method device provided in the embodiment of the present application can collect images of the detection object during the activity by using a camera, and after inputting the images into the deep learning model, it can obtain the corresponding current progress result and current prediction result in real time, and very efficiently and accurately master the current progress and predicted progress of the activity without having to go to the site to check.

[0102] In some embodiments, the processing module 902 is specifically used for: converting the first image to obtain a first image frame with a preset structure; the preset structure is related to the data input format of the deep learning model; converting the second image to obtain a second image frame with the preset structure; connecting the first image frame and the second image frame to obtain the current image frame with the preset structure.

[0103] In some embodiments, the processing module 902 is specifically configured to: input the first image into a convolutional neural network to obtain a first image frame with a preset structure.

[0104] In some embodiments, the processing module 902 is specifically configured to: construct a target three-dimensional solid according to the first image and the third image; a first dimension of the target three-dimensional solid is one dimension of the first image and the third image, a second dimension of the target three-dimensional solid is the other dimension of the second image and the third image, and a third dimension of the target three-dimensional solid is the time dimension where the current moment and the previous moment are located; divide the target three-dimensional solid to obtain a plurality of sub-solids; a dimension plane of the sub-solid is parallel or coincident with a dimension plane of the target three-dimensional solid; convert the second image into the target three-dimensional solid, and for each sub-solid, determine a value corresponding to the sub-solid based on data corresponding to the sub-solid in the second image; based on a preset rule, cascade the values corresponding to the plurality of sub-solids to obtain a second image frame.

[0105] In some embodiments, the input module 903 is specifically configured to: perform encoding processing, decoding processing, and fully connected processing on the current image frame in sequence to obtain a current progress result; determine a current prediction result according to the current progress result and a previous prediction result.

[0106] In some embodiments, the input module 903 is specifically configured to: perform encoding processing on the current image frame to obtain an encoding result; perform decoding processing on the encoding result and a previous output result to obtain a current output result; the previous output result is the current output result corresponding to the previous image frame; perform fully connected processing on the current output result to obtain a current progress result; determine a current prediction deviation according to the current progress result and a previous prediction result; perform fully connected processing after connecting the current output result and the current prediction deviation to obtain a current prediction result.

[0107] In some embodiments, the pushing module 904 is specifically configured to: obtain identification information of a detection object corresponding to the first image and the second image; generate an alarm message according to the prediction result; based on the identification information, push the current progress result, the current prediction result, and the alarm message to a target user.

[0108] The progress detection method device provided by the embodiments of the present application can be used to execute the above method embodiments, and its implementation principle and technical effects are similar, which will not be elaborated here in this embodiment.

[0109] Figure 10It is a schematic hardware structure diagram of the progress detection method device provided by the embodiments of the present application. The device can be a computer, a message transceiver device, a tablet device, a medical device, etc.

[0110] Device 100 may include one or more of the following components: a processing component 1001, a memory 1002, a power supply component 1003, a multimedia component 1004, an audio component 1005, an input / output (I / O) interface 1006, a sensor component 1007, and a communication component 1008.

[0111] The processing component 1001 generally controls the overall operation of the device 100, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 1001 may include one or more processors 1009 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 1001 may include one or more modules to facilitate the interaction between the processing component 1001 and other components. For example, the processing component 1001 may include a multimedia module to facilitate the interaction between the multimedia component 1004 and the processing component 1001.

[0112] The memory 1002 is configured to store various types of data to support the operation of the device 100. Examples of these data include instructions for any application or method operating on the device 100, contact data, phone book data, messages, pictures, videos, etc. The memory 1002 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk, or an optical disc.

[0113] The power supply component 1003 provides power to various components of the device 100. The power supply component 1003 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 100.

[0114] The multimedia component 1004 includes a screen that provides an output interface between the device 100 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 1004 includes a front camera and / or a rear camera. When the device 100 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

[0115] The audio component 1005 is configured to output and / or input audio signals. For example, the audio component 1005 includes a microphone (MIC) that is configured to receive external audio signals when the device 100 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 1002 or transmitted via the communication component 1008. In some embodiments, the audio component 1005 further includes a speaker for outputting audio signals.

[0116] The I / O interface 1006 provides an interface between the processing component 1001 and a peripheral interface module, which can be a keyboard, a click wheel, buttons, etc. These buttons can include but are not limited to: a home button, a volume button, a power button, and a lock button.

[0117] The sensor component 1007 includes one or more sensors for providing a status assessment of various aspects of the device 100. For example, the sensor component 1007 can detect the on / off state of the device 100, the relative positioning of components, such as the display and keypad of the device 100. The sensor component 1007 can also detect a change in the position of the device 100 or a component of the device 100, the presence or absence of user contact with the device 100, the orientation or acceleration / deceleration of the device 100, and the temperature change of the device 100. The sensor component 1007 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 1007 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 1007 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0118] The communication component 1008 is configured to facilitate communication between the device 100 and other devices in a wired or wireless manner. The device 100 can access a communication standard-based wireless network, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 1008 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1008 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0119] In an exemplary embodiment, the device 100 can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components for performing the above method.

[0120] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as the memory 1002 including instructions, and the above instructions can be executed by the processor 1009 of the device 100 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, Random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0121] The above computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0122] An exemplary readable storage medium is coupled to the processor so that the processor can read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0123] Those of ordinary skill in the art will understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0124] The embodiments of the present application also provide a computer program product, including a computer program, which, when executed by a processor, implements the progress detection method executed by the above progress detection method device.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A progress detection method, characterized in that, Including: Obtain a first image and a second image; The first image is obtained by a black-and-white camera at the current moment; The second image is obtained by an event camera between the current moment and the previous moment; the previous moment is the moment when the black-and-white camera captured a third image adjacent to the first image before the current moment; Preprocess the first image and the second image to obtain a current image frame; Input the current image frame into a deep learning model to obtain a current progress result and a current prediction result corresponding to the current image frame; Push the current progress result and the current prediction result to the target user; The preprocessing the first image and the second image to obtain a current image frame includes: Convert the first image to obtain a first image frame with a preset structure; the preset structure is related to the data input format of the deep learning model; Convert the second image to obtain a second image frame with the preset structure; Connect the first image frame and the second image frame to obtain a current image frame with the preset structure; The converting the second image to obtain a second image frame with a preset structure includes: Construct a target three-dimensional object corresponding to the second image according to the first image and the third image; the first dimension of the target three-dimensional object is one dimension of the first image and the third image, the second dimension of the target three-dimensional object is the other dimension of the second image and the third image, and the third dimension of the target three-dimensional object is the time dimension where the current moment and the previous moment are located; Slice the target three-dimensional object to obtain a plurality of sub-three-dimensional objects; the dimensional plane of the sub-three-dimensional object is parallel or coincident with the dimensional plane of the target three-dimensional object; Convert the second image into the target three-dimensional object, and for each sub-three-dimensional object, determine the value corresponding to the sub-three-dimensional object based on the data corresponding to the sub-three-dimensional object obtained in the second image; Based on a preset rule, cascade the values corresponding to the plurality of sub-three-dimensional objects to obtain a second image frame.

2. The method according to claim 1, characterized in that, The converting the first image to obtain a first image frame with a preset structure includes: Input the first image into a convolutional neural network to obtain a first image frame with a preset structure.

3. The method according to claim 1, wherein The inputting the current image frame into a deep learning model to obtain a current progress result and a current prediction result corresponding to the current image frame includes: Perform encoding processing, decoding processing, and fully connected processing on the current image frame in sequence to obtain a current progress result; Determine the current prediction result according to the current progress result and the previous prediction result.

4. The method according to claim 3, wherein The performing encoding processing, decoding processing, and fully connected processing on the current image frame in sequence to obtain a current progress result includes: Perform encoding processing on the current image frame to obtain an encoding result; Perform decoding processing on the encoding result and the previous output result to obtain a current output result; the previous output result is the current output result corresponding to the previous image frame; Perform fully connected processing on the current output result to obtain a current progress result; Determining the current prediction result according to the current progress result and the previous prediction result includes: Determining the current prediction deviation according to the current progress result and the previous prediction result; Performing a fully connected process after connecting the current output result and the current prediction deviation to obtain the current prediction result.

5. The method according to any one of claims 1-4, characterized in that, Pushing the current progress result and the current prediction result to the target user includes: Obtaining the identification information of the detection object corresponding to the first image and the second image; Generating an alarm message according to the prediction result; Based on the identification information, pushing the current progress result, the current prediction result and the alarm message to the target user.

6. A progress detection method and device, characterized in that, Including: A black-and-white camera, an event camera, at least one processor and a memory; The black-and-white camera is configured to capture a first image and send the first image to the at least one processor; The event camera is configured to capture a second image and send the second image to the at least one processor; The memory stores computer execution instructions; The at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the progress detection method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, Computer execution instructions are stored in the computer-readable storage medium, and when the processor executes the computer execution instructions, the progress detection method according to any one of claims 1 to 5 is implemented.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, the progress detection method according to any one of claims 1 to 5 is implemented.

Images