Data processing method and electronic device

Abstract

The embodiment of the present application provides a data processing method and electronic device, which relates to the field of data processing technology and is applied to electronic devices. The electronic device is a back-end device for calling a second analysis model. The method includes: receiving code stream data and initial analysis results reported by a front-end device; determining frame identification information of multiple target video frames belonging to the target code stream based on the alarm description information and the object description information of each object; decoding the multiple target video frames from the target code stream based on the determined frame identification information to obtain an image sequence corresponding to the alarm object; calling the second analysis model to perform alarm analysis processing on the image sequence to obtain a target analysis result corresponding to the alarm object; wherein the target analysis result is used to characterize: retaining or eliminating the alarm object. This solution can reduce the underreporting of alarm analysis.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and electronic equipment. Background Art

[0002] With the popularization of camera equipment, the method of issuing alarms based on image data collected by the camera equipment has become more and more common in daily life.

[0003] In the related art, a second analysis model and a first analysis model are established for alarm analysis of image data. The first analysis model located in the front-end device is used to identify the object of the image data, and it is determined whether to alarm the identified object to obtain the alarm result of the first analysis model. Then, the alarm result of the first analysis model is secondary analyzed by the second analysis model located in the back-end device, and the false alarms of the first analysis model are eliminated to improve the accuracy of the alarm. Among them, the second analysis model and the first analysis model have the ability to identify the same type of alarm events (one alarm event can indicate that there is an alarm object, and multiple alarm events of the same type can generate alarms for different alarm objects), and the accuracy of the second analysis model is greater than that of the first analysis model.

[0004] In the alarm analysis process, the first analysis model generally performs alarm analysis on the continuous image sequence collected by the camera device, and the second analysis model performs secondary analysis on the image that generates an alarm after the analysis by the first analysis model. In the process of implementing this application, the inventor found that compared with the image data analyzed by the first analysis model, the image data analyzed by the second analysis model lost part of the data. After the secondary analysis, some alarms may be mistakenly eliminated, which may eventually result in missed alarms. Summary of the invention

[0005] The purpose of the embodiment of the present application is to provide a data processing method and electronic device to reduce the underreporting of alarm analysis. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to an electronic device, wherein the electronic device is a backend device for calling a second analysis model, and the method includes:

[0007] Receive the bitstream data and initial analysis results reported by the front-end device; wherein the bitstream data includes data obtained by encapsulating the target bitstream and the intermediate analysis results by the front-end device; the intermediate analysis results include object description information of each object identified by the front-end device calling the first analysis model to perform alarm analysis on the target bitstream; the initial analysis results include alarm description information of the alarm object obtained by the alarm analysis;

[0008] Based on the alarm description information and the object description information of each object, frame identification information of multiple target video frames belonging to the target code stream is determined; wherein the multiple target video frames contain the alarm objects indicated by the alarm description information, and time information corresponding to the multiple target video frames is no later than time information corresponding to the alarm video frames indicated by the alarm description information;

[0009] Based on the determined frame identification information, the plurality of target video frames are decoded from the target bitstream to obtain an image sequence corresponding to the alarm object;

[0010] The second analysis model is called to perform alarm analysis processing on the image sequence to obtain a target analysis result corresponding to the alarm object; wherein the target analysis result is used to indicate whether to retain or eliminate the alarm object.

[0011] In a second aspect, an embodiment of the present application provides an electronic device, including:

[0012] Memory, used to store computer programs;

[0013] The processor is used to implement any of the above-mentioned data processing methods when executing the program stored in the memory.

[0014] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned data processing methods.

[0015] Beneficial effects of the embodiments of the present application:

[0016] The data processing method provided in the embodiment of the present application is applied to a back-end device that calls a second analysis model, and can receive the code stream data and the initial analysis result reported by the front-end device; the code stream data includes the target code stream used by the front-end device for alarm analysis, and the intermediate analysis result of the object description information of each object identified by the alarm analysis of the target code stream, and the initial analysis result includes the alarm description information of the alarm object obtained by the alarm analysis; the electronic device can determine the frame identification information of multiple target video frames belonging to the target code stream based on the alarm description information and the object description information of each object, the multiple target video frames contain the alarm object indicated by the alarm description information, and the time information corresponding to the multiple target video frames is not later than the time information corresponding to the alarm video frame of the alarm object; thereafter, based on the frame identification information, multiple target video frames can be decoded from the target code stream to obtain an image sequence corresponding to the alarm object, and the second analysis model can be called to perform alarm analysis processing on the image sequence to obtain a target analysis result for retaining or eliminating the alarm object. In the present application, multiple target video frames obtained based on the analysis of the alarm description information and the object description information contain an alarm object, and the time information of the multiple target video frames is no later than the time information corresponding to the alarm video frame. In this way, the image sequence of the multiple target video frames can characterize the rich object information of the alarm object, such as behavior route information. Therefore, by performing alarm analysis processing on the image sequence through the second analysis model, the alarm object can be accurately retained or eliminated compared to the image that generates an alarm reported only based on the first analysis model, so as to reduce underreporting of the alarm analysis.

[0017] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0019] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a secondary analysis process of a second analysis model provided in an embodiment of the present application;

[0021] Figure 3 Another schematic diagram of the secondary analysis process of the second analysis model provided in an embodiment of the present application;

[0022] Figure 4A schematic diagram of a flow chart of a method for selecting an image sequence provided in an embodiment of the present application;

[0023] Figure 5a A schematic diagram of a process for saving I frame information provided in an embodiment of the present application;

[0024] Figure 5b A schematic diagram of a process for selecting a target segment provided in an embodiment of the present application;

[0025] Figure 6 Another flowchart of a data processing method provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a flow chart of calling a second analysis model to perform alarm analysis on an image sequence provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of a process for determining the absolute time of a front end provided in an embodiment of the present application;

[0028] Fig. 9 A block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0030] In related technologies, although the second analysis model has greater representation and learning capabilities, it performs poorly in some sub-domains (specific fields), and retraining or iteration consumes a lot of resources. Although the first analysis model can be trained quickly, its generalization performance is unsatisfactory, and its accuracy fluctuates greatly with changes in objects or environments, requiring frequent manual iterations.

[0031] In order to solve this problem, the method of cooperating the second analysis model and the first analysis model is generally adopted: a first analysis model is used at the front end. Generally, the performance of this first analysis model is average, but the analysis efficiency is high. It can quickly analyze the objects in the video and generate alarms, but the accuracy of the alarms is low; a second analysis model is used at the back end. The second analysis model has higher accuracy and analyzes the alarm results of the objects analyzed by the first analysis model at the front end again.

[0032] However, the following problems may occur during the collaboration between the second analysis model and the first analysis model:

[0033] At present, the first analysis model generally uses a continuous image sequence for analysis, while the second analysis model uses the alarm image generated by the first analysis model for further analysis. Obviously, since only the image that generates the alarm is analyzed, the second analysis model loses some information during the analysis. For example: if the alarm image of the first analysis model is occluded, the front-end first analysis model can perform alarm analysis based on the same object without occlusion in an image sequence, while the back-end second analysis model cannot perform effective recognition (the second analysis model can only use the image that generates the alarm by the first analysis model for alarm analysis).

[0034] In order to solve the problem of partial loss of image data analyzed by the second analysis model, the first analysis model is currently customized to output a copy of the analysis results of the first analysis model (such as image data a few seconds or frames before the alarm). This method solves the problem of partial data loss to a certain extent, but the data obtained depends on the customization of the first analysis model at the front end (the analysis results of the customized first analysis model may not necessarily include the alarm object). At the same time, the front-end device usually does not support customization due to its old version, and needs to be upgraded before customization can be performed.

[0035] In addition, the previous solution of video stream analysis required searching for files from the video files and reading files through IO (input / output), which was inefficient. In addition, if multiple analysis channels were opened, a larger memory space would need to be opened for each additional analysis channel if pre-reading was performed, which consumed a large amount of memory resources.

[0036] Based on this, the embodiment of the present application provides a data processing method and electronic device, which can accurately obtain more data based on the video stream and provide it to the second analysis model of the back-end device, which can improve the effect of the secondary analysis, and this method is based on the existing technology and is applicable to a wider range of front-end and back-end collaboration. Different from the existing technology, the present application can filter out an image sequence of video frames containing the alarm object to be analyzed again from the bit stream to provide it to the back-end for collaborative analysis, which can effectively solve the pain points of the existing technology such as the target being blocked and the need to customize the front-end program, and reduce the underreporting of the alarm analysis.

[0037] The following first introduces a data processing method provided in an embodiment of the present application.

[0038] Among them, a data processing method provided in an embodiment of the present application can be applied to an electronic device, and the electronic device can be a back-end device for calling a second analysis model, and can be a video recorder or a terminal device, etc. The terminal device can be a mobile phone or a computer, etc. The present application does not limit the specific form of the electronic device. A data processing method provided in an embodiment of the present application can be applied to a scenario in which a front-end first analysis model and a back-end second analysis model collaborate to perform alarm analysis on image data. Among them, the front-end first analysis model is the first analysis model located at the front-end device, and the back-end second analysis model is the second analysis model located at the back-end device.

[0039] In addition, in a data processing method provided in an embodiment of the present application, the front-end device and the back-end device may be located in the same device or different devices. For example, the front-end device may be a video camera, etc., which may call a first analysis model to perform alarm analysis on the image data of the video collected by it, and report the alarm image data, etc. to the back-end device. The back-end device may be a video recorder or a terminal device, which may receive the image data, etc. reported by the front-end device for secondary analysis. Alternatively, the front-end device and the back-end device may be located in the same device, the front-end device may be a first slave processor in the electronic device, and the back-end device may be a second slave processor in the electronic device. The electronic device may also include a main processor. The main processor may interact with the first slave processor and the second slave processor respectively to achieve the purpose of eliminating false alarms and reducing missed alarms in the analysis results of the image data.

[0040] A data processing method provided in an embodiment of the present application is applied to an electronic device, where the electronic device is a backend device for calling a second analysis model. The method includes:

[0041] Receive the bitstream data and initial analysis results reported by the front-end device; wherein the bitstream data includes data obtained by encapsulating the target bitstream and the intermediate analysis results by the front-end device; the intermediate analysis results include object description information of each object identified by the front-end device calling the first analysis model to perform alarm analysis on the target bitstream; the initial analysis results include alarm description information of the alarm object obtained by the alarm analysis;

[0042] Based on the alarm description information and the object description information of each object, frame identification information of multiple target video frames belonging to the target code stream is determined; wherein the multiple target video frames contain the alarm objects indicated by the alarm description information, and time information corresponding to the multiple target video frames is no later than time information corresponding to the alarm video frames indicated by the alarm description information;

[0043] Based on the determined frame identification information, the plurality of target video frames are decoded from the target bitstream to obtain an image sequence corresponding to the alarm object;

[0044] The second analysis model is called to perform alarm analysis processing on the image sequence to obtain a target analysis result corresponding to the alarm object; wherein the target analysis result is used to indicate whether to retain or eliminate the alarm object.

[0045] The data processing method provided in the embodiment of the present application is applied to a back-end device that calls a second analysis model, and can receive the code stream data and the initial analysis result reported by the front-end device; the code stream data includes the target code stream used by the front-end device for alarm analysis, and the intermediate analysis result of the object description information of each object identified by the alarm analysis of the target code stream, and the initial analysis result includes the alarm description information of the alarm object obtained by the alarm analysis; the electronic device can determine the frame identification information of multiple target video frames belonging to the target code stream based on the alarm description information and the object description information of each object, the multiple target video frames contain the alarm object indicated by the alarm description information, and the time information corresponding to the multiple target video frames is not later than the time information corresponding to the alarm video frame of the alarm object; thereafter, based on the frame identification information, multiple target video frames can be decoded from the target code stream to obtain an image sequence corresponding to the alarm object, and the second analysis model can be called to perform alarm analysis processing on the image sequence to obtain a target analysis result for retaining or eliminating the alarm object. In the present application, multiple target video frames obtained based on the analysis of the alarm description information and the object description information contain an alarm object, and the time information of the multiple target video frames is no later than the time information corresponding to the alarm video frame. In this way, the image sequence of the multiple target video frames can characterize the rich object information of the alarm object, such as behavior route information. Therefore, by performing alarm analysis processing on the image sequence through the second analysis model, the alarm object can be accurately retained or eliminated compared to the image that generates an alarm reported only based on the first analysis model, so as to reduce underreporting of the alarm analysis.

[0046] A data processing method provided in an embodiment of the present application is exemplarily introduced below in conjunction with the accompanying drawings.

[0047] like Figure 1 As shown, a data processing method provided in an embodiment of the present application is applied to an electronic device, where the electronic device is a backend device for calling a second analysis model, and may include the following steps:

[0048] S101: receiving the bitstream data and initial analysis results reported by the front-end device;

[0049] The code stream data includes data obtained by encapsulating the target code stream and the intermediate analysis result by the front-end device; the intermediate analysis result includes object description information of each object identified by the front-end device calling the first analysis model to perform alarm analysis on the target code stream; the initial analysis result includes alarm description information of the alarm object obtained by the alarm analysis;

[0050] In a data processing method provided by an embodiment of the present application, a front-end device can superimpose the object description information of each object identified by the front-end first analysis model when performing alarm analysis on the target code stream into the target code stream, that is, the code stream data reported by the front-end device includes the target code stream and the intermediate analysis result; of course, the front-end device can also report the initial analysis result obtained by performing alarm analysis on the target code stream, and the initial analysis result includes the alarm description information of the alarm object. Among them, the alarm description information of the alarm object may include: the ID of the alarm object (the ID of the object that triggers the alarm event), the alarm frame (that is, the position frame of the alarm object), the alarm time of the alarm object, of course, an alarm object can belong to an alarm event, and the alarm event can have alarm event description information, which can include the ID of the alarm event, the alarm type, etc. for describing the content of the alarm event, and the alarm type can include: crossing the boundary, regional invasion, etc., which are not limited here; and the object description information of each object can include: object ID, object identification frame (that is, object location frame), time of identifying the object (that is, time information of the video frame of the object being identified), etc. Due to the movement of the object, an object ID can correspond to different times of identifying the object, and different identification frames. The process of the front-end device performing alarm analysis on the target code stream may be similar to the prior art and will not be described in detail here.

[0051] In addition, the front-end device can report the bitstream data and the initial analysis result to the back-end device at the same time, or the front-end device can report the bitstream data and the initial analysis result to the back-end device in sequence, and the reporting order is not limited, which is reasonable.

[0052] It should be noted that the first analysis model can also be called a small model, and the second analysis model can also be called a large model. There are differences between the large model and the small model in terms of training data volume, memory occupied, accuracy, analysis speed, etc.; for example, the framework of the small model is different from that of the large model, such as the number of hierarchical structures of the large model is greater than the number of hierarchical structures of the small model. The large model can be a neural network model with a large number of parameters (such as millions or billions of parameters) that can perform complex processing on large-scale data, and the small model is a neural network model with fewer parameters that can process small amounts of data; the memory occupied by the small model is less than the memory occupied by the large model; the analysis speed of the small model is greater than the analysis speed of the large model. The small model can support video analysis tasks or image analysis tasks with higher analysis speed requirements, while the large model usually supports image analysis tasks, or supports video analysis tasks with lower analysis speed requirements.

[0053] Moreover, the large model and the small model have the ability to identify the same type of alarm events, but the number of types of alarm events that the large model can identify is greater than the number of types of alarm events that the small model can identify. For example, the small model has the ability to identify animal alarm events, and the large model has at least the ability to identify animal and plant alarm events.

[0054] S102: Determine frame identification information of a plurality of target video frames belonging to the target bitstream based on the alarm description information and the object description information of each object;

[0055] The multiple target video frames include the alarm object indicated by the alarm description information, and the time information corresponding to the multiple target video frames is no later than the time information corresponding to the alarm video frame indicated by the alarm description information;

[0056] After receiving the code stream data and the initial analysis results, for each alarm object, first determine the frame identification information of multiple target video frames belonging to the target code stream based on the alarm description information of the alarm object and the object description information of each object, each of the multiple target video frames contains the alarm object indicated by the alarm description information, and the corresponding time information is no later than the time information corresponding to the video frame indicated by the alarm description information, that is, the multiple target video frames can represent more object information of the alarm object before the alarm; subsequently, based on the frame identification information, determine the image sequence corresponding to the alarm object indicated by the alarm description information, and provide more image data for the second analysis model to perform a secondary alarm analysis on the alarm object. Among them, multiple target video frames can be arranged in chronological order, and the frame identification information of the multiple target video frames includes: frame identifications corresponding to each of the multiple target video frames arranged in sequence; and, in the target code stream, the multiple target video frames may not be directly adjacent to each other, for example: the multiple target video frames are the 1st, 7th, 12th, 19th, 23rd, 26th, 30th, 34th, 38th, 40th, 45th, and 50th frames in the target code stream. Of course, the multiple target video frames may also be directly adjacent, and this application does not limit this.

[0057] Exemplarily, in one implementation, the object description information of each object may be first screened, and then the screened object description information may be used to determine the frame identification information, wherein the frame identification information of multiple target video frames belonging to the target code stream is determined based on the alarm description information and the object description information of each object, including:

[0058] Based on the alarm time information in the alarm description information, each first object description information is selected from the object description information of each object; wherein, in each first object description information, the time represented by the time information about the object is within a time range having a second duration and ending at the time represented by the alarm time information; the time information of any object is the time information of the video frame in which the object is identified;

[0059] From each of the first object description information, select the object description information in which the indicated object and the alarm object in the alarm description information are the same object, to obtain the second object description information;

[0060] Based on the time information corresponding to the object in the obtained second object description information, frame identification information of multiple target video frames belonging to the target code stream is determined.

[0061] When determining the frame identification information of multiple target video frames, each first object description information can be selected from the object description information of each object based on the alarm time information in the alarm description information, that is, first based on the alarm time information in the alarm description information, the object description information of each object is screened to obtain each first object description information having a second time range and an end time represented by the alarm time information; then, from each first object description information, the object description information of the indicated object and the alarm object in the alarm description information can be selected to obtain the second object description information, for example: the ID of the alarm object in the alarm description information is matched with the object ID corresponding to each of the first object description information, and the first object description information to which the matched object ID belongs is used as the second object description information; in the embodiment of the present application, each object description information contains time information corresponding to the object, that is, the time information of any object is the time information of the video frame from which the object is identified, and the time information represents the video frame to which the object belongs when it is identified, and the frame identification information of multiple target video frames belonging to the target code stream can be determined based on the time information corresponding to the object in the second object description information. Since the time information of any object is the time information of the video frame in which the object is identified, the video frame in which the object is identified is the target video frame. The time information of each video frame in which the object is identified can be used to obtain frame identification information of multiple target video frames.

[0062] Among them, the second duration can be any preset duration, and this application does not limit this; after actual application, in order to ensure that the objects represented by each first object description information can include the alarm object and include the unobstructed alarm object, the second duration can be 5s.

[0063] In addition, when determining the frame identification information of multiple target video frames belonging to the target code stream, you can first select the object description information of each object, in which the indicated object and the alarm object in the alarm description information are the same object, to obtain the third object description information; then, based on the alarm time information in the alarm description information, select the fourth object description information from the third object description information; in the fourth object description information, the time represented by the time information about the object is within a time range with the time represented by the alarm time information as the end time and having a second duration.

[0064] That is, in an embodiment of the present application, first, based on the alarm time information, from the object description information of each object, first object description information with a second duration range can be screened, and then the second object description information of the same object as the alarm object can be selected, and the frame identification information of multiple target video frames belonging to the target code stream can be determined according to the time information corresponding to the object in the second object description information; or, first, from the object description information of each object, third object description information of the same object as the alarm object can be screened, and then, based on the alarm time information, fourth object description information with a second duration can be selected from the third object description information, and the frame identification information of multiple target video frames belonging to the target code stream can be determined according to the time information corresponding to the object in the fourth object description information. The present application does not limit the order of screening by time and screening by alarm object. The second object description information and the fourth object description information can be the same.

[0065] S103: Based on the determined frame identification information, the plurality of target video frames are decoded from the target bitstream to obtain an image sequence corresponding to the alarm object;

[0066] After the frame identification information of the multiple target video frames is determined, the multiple target video frames can be decoded from the target bitstream based on the determined frame identification information to obtain an image sequence corresponding to the alarm object of the alarm object.

[0067] In order to facilitate decoding of multiple target video frames from the target code stream, and considering the performance of the back-end device (directly analyzing the entire target code stream requires allocating more resources, affecting the performance), the target code stream can also be screened; illustratively, based on the determined frame identification information, the multiple target video frames are decoded from the target code stream to obtain the image sequence corresponding to the alarm object, and the method further includes:

[0068] Determine a designated code stream segment in the target code stream; wherein the designated code stream segment is a code stream segment having a first duration and having an end time represented by the alarm time information in the alarm description information;

[0069] The step of decoding the plurality of target video frames from the target bitstream based on the determined frame identification information to obtain an image sequence corresponding to the alarm object includes:

[0070] Determine a target segment in the designated bitstream segment that includes the video frame indicated by the frame identification information;

[0071] The multiple target video frames are decoded from the target segment to obtain an image sequence corresponding to the alarm object.

[0072] The present application is to provide the second analysis model with an image sequence of more object information of the alarm object that is no later than the alarm time of the alarm object, so that the second analysis model can analyze the alarm object based on more image data, and when filtering the target code stream, it can determine the specified code stream segment in the target code stream that has the time represented by the alarm time information as the end time and has a first duration.

[0073] Accordingly, when decoding to obtain an image sequence, a target segment in a specified bitstream segment that includes multiple target video frames indicated by the frame identification information can be determined, and then multiple target video frames can be decoded from the target segment to obtain an image sequence corresponding to the alarm object. Among them, the multiple target video frames are arranged in chronological order, the start time of the target segment is not later than the first target video frame among the multiple target video frames, and the end time of the target segment is not earlier than the last target video frame among the multiple target video frames; after obtaining the target segment, multiple target video frames can be selected from the target segment based on the frame identification information to obtain an image sequence corresponding to the alarm object.

[0074] In addition, the present application determines the frame identification information of multiple target video frames of the target code stream based on the second object description information of the second duration, and then parses multiple target video frames from the specified code stream segment of the first duration. Usually, the first duration and the second duration can correspond to each other, for example: the first duration and the second duration are both 3s; in order to find the video frame of the unobstructed alarm object, the second object description information can have a longer time, that is, the second duration can be greater than the first duration, for example: the second duration is 5s and the first duration is 3s; of course, in order to make the specified code stream segment fully contain multiple target video frames, the first duration may also be greater than the second duration, for example: the first duration is 5s and the second duration is 3s; the first duration and the second duration can be adjusted according to actual needs, and the present application does not limit this.

[0075] S104: calling the second analysis model to perform alarm analysis processing on the image sequence to obtain a target analysis result corresponding to the alarm object;

[0076] The target analysis result is used to indicate whether to retain or eliminate the alarm object.

[0077] After obtaining the image sequence corresponding to the alarm object, the second analysis model can be called to perform alarm analysis on the image sequence to obtain the target analysis result corresponding to retaining or eliminating the alarm object; since the alarm analysis is performed by utilizing the image sequence of more object information of the alarm object before the alarm, the alarm object can be accurately retained or eliminated, reducing the underreporting of the alarm analysis.

[0078] Exemplarily, in one implementation, the calling of the second analysis model to perform alarm analysis processing on the image sequence to obtain a target analysis result corresponding to the alarm object includes:

[0079] Dividing the image sequence into a plurality of image groups according to time sequence;

[0080] A target image group among the multiple image groups is used as the current image group; wherein the time represented by the time information of the video frames in the target image group is closest to the time represented by the alarm time information in the alarm description information relative to other image groups;

[0081] Inputting the current image group into the second analysis model, so that the second analysis model performs alarm analysis on the current image group to obtain a secondary analysis result corresponding to the current image group;

[0082] In response to the secondary analysis result corresponding to the current image group indicating the existence of the alarm object, retaining the alarm object as the target analysis result of the alarm object;

[0083] In response to the secondary analysis result corresponding to the current image group indicating that the alarm object does not exist, if there is a previous image group for the current image group, the previous image group is used as the current image group, and the step of inputting the current image group into the second analysis model is returned so that the second analysis model performs alarm analysis on the current image group to obtain the secondary analysis result corresponding to the current image group; if there is no previous image group for the current image group, the alarm object is eliminated as the target analysis result of the alarm object.

[0084] It can be understood that the closer the image in the image sequence is to the time represented by the alarm time information, the more accurately it can represent the state of the alarm object. The present application can divide the image sequence into multiple image groups in chronological order, and use the target image group closest to the alarm time as the current image group; the current image group is input into the second analysis model for alarm analysis to obtain the secondary analysis result corresponding to the current image group. The secondary analysis result can represent whether the alarm object exists. For example, if the second analysis model also alarms the alarm object, then the alarm object exists. If the second analysis model does not alarm the alarm object, then there is no alarm for the alarm object. If the secondary analysis result corresponding to the current image group represents the existence of the alarm object, that is, the analysis results of the second analysis model and the first analysis model are consistent, then it means that the alarm analysis of the first analysis model for the alarm object is accurate. If it is correct, the alarm object will be retained as the target analysis result of the alarm object; if the secondary analysis result corresponding to the current image group indicates that the alarm object does not exist, that is, the analysis results of the second analysis model and the first analysis model are inconsistent, then the alarm object may be obstructed in the image analyzed by the second analysis model for the alarm object, and the search can continue for the image in which the alarm object is not obstructed. When the previous image group exists in the current image group, the previous image group is used as the new current image group, and the alarm analysis is re-performed on the new current image group until all image groups are analyzed; if the previous image group does not exist in the current image group, that is, all image groups are analyzed, and the corresponding secondary analysis result indicates that there is no alarm for the alarm object, then the alarm of the first analysis model for the alarm object is a false alarm, and the alarm object will be eliminated as the target analysis result of the alarm object.

[0085] In addition, each of the multiple image groups can be directly input into the second analysis model, and the second analysis model performs alarm analysis on each image group to obtain the secondary analysis result corresponding to each image group; then, the target analysis result of retaining or eliminating the alarm object is obtained by combining the secondary analysis result of each image group. For example, if the multiple image groups are 4 image groups, the secondary analysis results of three image groups all indicate the existence of the alarm object, and the secondary analysis result of one image group indicates the absence of the alarm of the alarm object, then the target analysis result obtained can be: retain the alarm object.

[0086] It should be noted that the above-mentioned method of calling the second analysis model to perform alarm analysis on the image sequence and obtaining the target analysis result corresponding to the alarm object is merely an example. Any method that can call the second analysis model to perform alarm analysis on the image sequence and obtain the target analysis result of retaining or eliminating the alarm object is applicable to the present application, and the present application does not limit this.

[0087] In the technical solution of the present application, the acquisition, storage, use, processing, transmission, provision and disclosure of the code stream data, analysis results, frame identification information and image sequences involved are all carried out with the user's authorization.

[0088] The data processing method provided in the embodiment of the present application is applied to a back-end device that calls a second analysis model, and can receive the code stream data and the initial analysis result reported by the front-end device; the code stream data includes the target code stream used by the front-end device for alarm analysis, and the intermediate analysis result of the object description information of each object identified by the alarm analysis of the target code stream, and the initial analysis result includes the alarm description information of the alarm object obtained by the alarm analysis; the electronic device can determine the frame identification information of multiple target video frames belonging to the target code stream based on the alarm description information and the object description information of each object, the multiple target video frames contain the alarm object indicated by the alarm description information, and the time information corresponding to the multiple target video frames is not later than the time information corresponding to the alarm video frame of the alarm object; thereafter, based on the frame identification information, multiple target video frames can be decoded from the target code stream to obtain an image sequence corresponding to the alarm object, and the second analysis model can be called to perform alarm analysis processing on the image sequence to obtain a target analysis result for retaining or eliminating the alarm object. In the present application, multiple target video frames obtained based on the analysis of the alarm description information and the object description information contain an alarm object, and the time information of the multiple target video frames is no later than the time information corresponding to the alarm video frame. In this way, the image sequence of the multiple target video frames can characterize the rich object information of the alarm object, such as behavior route information. Therefore, by performing alarm analysis processing on the image sequence through the second analysis model, the alarm object can be accurately retained or eliminated compared to the image that generates an alarm reported only based on the first analysis model, so as to reduce underreporting of the alarm analysis.

[0089] Optionally, in another embodiment of the present application, the alarm time information in the alarm description information is: the front-end absolute time when the front-end device recognizes the alarm object; the time information about the object in the object description information of each object includes: the relative time of the video frame in which the object is recognized;

[0090] Before selecting each first object description information from the object description information of each object based on the alarm time information in the alarm description information, the method further includes:

[0091] Determine the front-end absolute time corresponding to the relative time of each object;

[0092] The selecting each first object description information from the object description information of each object based on the alarm time information in the alarm description information includes:

[0093] Based on the front-end absolute time of the alarm object and the front-end absolute time of each object, each first object description information is selected from the object description information of each object.

[0094] The bitstream data reported by the front-end device and the time information in the initial analysis results are all time information recorded by the front-end device, and each video frame in the front-end device has a relative time, and only the I frame (I frame, also known as the intra picture) has an absolute time; the alarm time information in the alarm description information is the front-end absolute time when the front-end device recognizes the alarm object; the time information about the object in the object description information of each object includes the relative time of the video frame in which the object is recognized; among them, the relative time of any object can represent the video frame when the object is recognized, the absolute time is the world time, and the front-end absolute time is the world time of the front-end device.

[0095] When selecting each first object description information, the back-end device needs to use the alarm time information and the front-end absolute time of the object description information of each object in the front-end device, so as to filter the object description information of each object according to the front-end absolute time to obtain each first object description information. In order to accurately obtain the image sequence corresponding to the alarm object later, the back-end device can determine the front-end absolute time corresponding to the relative time of each object before selecting each first object description information.

[0096] After determining the front-end absolute time of each object, each first object description information can be selected from the object description information of each object based on the front-end absolute time of the alarm object and the front-end absolute time of each object. For example, from the front-end absolute time of each object, object description information within a time range with the front-end absolute time of the alarm object as the end time and a second duration can be determined as each first object description information.

[0097] In addition, when determining the front-end absolute time corresponding to the relative time of each object, the absolute time of the target I frame recorded by the front-end device and the time difference between the relative time of the target I frame and the relative time of the object can be used to calculate the front-end absolute time corresponding to the relative time of the object. For example, the front-end absolute time of the target I frame and the time difference between the relative time of the target I frame and the relative time of the object are summed as the front-end absolute time corresponding to the relative time of the object; wherein, the target I frame is: an I frame whose relative time is closest to the relative time of the object and is before the relative time of the object.

[0098] The time difference between the relative time of the target I frame and the relative time of the object may be: the time difference between the front-end relative time of the target I frame and the front-end relative time of the object, or the time difference between the back-end relative time of the target I frame and the back-end relative time of the object. In practical applications, since the front-end device reports the code stream data and the initial analysis results to the back-end device, the back-end device may not be able to obtain the front-end relative time of the target I frame and the front-end relative time of the object (may be lost), considering that when setting the relative time for the target I frame and any other video frame that uses the target I frame as a decoding reference, the relative time difference used by the front-end device and the back-end device is the same, therefore, the time difference between the relative time of the target I frame and the relative time of the object may be: the time difference between the back-end relative time of the target I frame and the back-end relative time of the object.

[0099] It should be noted that each object may have multiple relative times about the object, and each relative time can be calculated in the above manner to obtain the corresponding front-end absolute time, which is not limited in this application.

[0100] It can be seen that in an embodiment of the present application, before selecting each first object description information from the object description information of each object, the front-end absolute time corresponding to the relative time of each object can also be determined, so as to accurately select each first object description information based on the front-end absolute time of the alarm object and the front-end absolute time of each object.

[0101] Optionally, in another embodiment of the present application, determining a target segment in the designated code stream segment that includes the video frame indicated by the frame identification information includes:

[0102] Determine an initial timestamp range according to the frame identification information of the multiple target video frames; wherein the starting timestamp of the initial timestamp range is the local timestamp of the first frame of the multiple target video frames, and the ending timestamp is the local timestamp of the last frame of the multiple target video frames; the local timestamp of any frame is the absolute time with respect to the local clock of the electronic device;

[0103] According to the initial timestamp range, a target segment in the designated code stream segment that includes the video frame indicated by the frame identification information is identified.

[0104] When determining the target segment, the target segment is a continuous segment in the specified code stream segment that includes multiple target video frames indicated by frame identification information. The initial timestamp range including the multiple target video frames can be determined based on the frame identification information of the multiple target video frames; and based on the initial timestamp range, the target segment in the specified code stream segment that includes the multiple target video frames indicated by the frame identification information is identified.

[0105] Among them, the starting timestamp of the initial timestamp range is the local timestamp of the first frame among the multiple target video frames, and the ending timestamp is the local timestamp of the last frame among the multiple target video frames. Since the multiple target video frames are arranged in chronological order, the initial timestamp range formed by the local timestamp of the first frame and the local timestamp of the last frame can include multiple target video frames.

[0106] It should be noted that the local timestamp of any video frame can be understood as the back-end absolute time of the video frame. For any video frame, if the video frame is encoded as an I frame, the back-end device can record the back-end absolute time and the back-end relative time of the video frame when receiving the video frame. The back-end relative time is obtained based on the conversion relationship between the front-end device and the back-end device regarding relative time; if the video frame is not encoded as an I frame, the back-end device can record the back-end relative time of the video frame based on the front-end relative time of the video frame and the above-mentioned conversion relationship.

[0107] It can be seen that for any video frame, if the video frame is an I frame, the back-end absolute time can be obtained from the recording information. If the video frame is not an I frame, the back-end absolute time of the video frame can be calculated based on the back-end absolute time and the back-end relative time of the specified I frame, and the back-end relative time of the video frame (the back-end relative time of the video frame can be obtained based on the front-end relative time of the video frame and the conversion relationship between the front-end device and the back-end device about relative time). Among them, the specified I frame is the I frame whose relative time is closest to the relative time of the video frame and is located before the relative time of the video frame among all I frames.

[0108] In the present application, multiplexing of the code stream buffer area is supported without the need to allocate additional memory space. Exemplarily, the method further includes:

[0109] In response to receiving the bitstream data, buffering a target bitstream in the bitstream data into a target buffer area, and recording a local timestamp of an I frame of the target bitstream;

[0110] The step of identifying, according to the initial timestamp range, a target segment in the designated bitstream segment that includes the video frame indicated by the frame identification information comprises:

[0111] Determine a target start timestamp according to an end timestamp in the initial timestamp range and a local timestamp of each I frame of the specified code stream segment; wherein the target start timestamp is a timestamp of an I frame, and the target start timestamp is smaller than a start timestamp in the initial timestamp range;

[0112] A segment of the designated code stream segment that is within a local timestamp range formed by the end timestamp and the target start timestamp is determined as a target segment.

[0113] After receiving the bitstream data, the backend device may cache the target bitstream in the bitstream data to the target cache area, and may record the local timestamp of each I frame in the target bitstream to facilitate the identification of subsequent target segments.

[0114] Correspondingly, when identifying the target segment according to the initial timestamp range, the target start timestamp that is smaller than the start timestamp within the initial timestamp range and is an I frame can be determined according to the end timestamp within the initial timestamp range and the local timestamps of each I frame of the specified code stream segment; then, the segment in the specified code stream segment that is within the local timestamp range formed by the end timestamp and the target start timestamp is determined as the target segment for subsequent decoding.

[0115] Optionally, determining the target start timestamp according to the end timestamp within the initial timestamp range and the local timestamp of each I frame of the designated code stream segment includes:

[0116] Determine the local timestamp of the first I frame in the specified code stream segment that is located before the end timestamp of the initial timestamp range as the current timestamp;

[0117] Identify the magnitude relationship between the current timestamp and the starting timestamp within the initial timestamp range;

[0118] In response to the current timestamp being less than the starting timestamp within the initial timestamp range, taking the current timestamp as the target starting timestamp;

[0119] In response to the current timestamp being not less than the starting timestamp within the initial timestamp range, the timestamp of the previous I frame of the current timestamp is used as the new current timestamp, and the step of identifying the size relationship between the current timestamp and the starting timestamp within the initial timestamp range is returned.

[0120] When determining the target start timestamp, in order to make the target start timestamp the local timestamp of the I frame, the local timestamp of the first I frame in the specified code stream segment and before the end timestamp of the initial timestamp range can be used as the current timestamp; in order to ensure that the target timestamp is less than the start timestamp in the initial timestamp range, continue to identify the size relationship between the current timestamp and the start timestamp in the initial timestamp range; if the current timestamp is less than the start timestamp, the I frame meets the conditions, and the current timestamp is directly used as the target start timestamp; if the current timestamp is not less than the start timestamp, the local timestamp of the previous I frame of the current timestamp is used as the new current timestamp, and continue to identify the size relationship between the current timestamp and the start timestamp until it is identified that the current timestamp is less than the start timestamp, and the target start timestamp is obtained.

[0121] The embodiment of the present application can identify whether the timestamp of the I frame is less than the start timestamp based on the end timestamp, so as to determine the target segment with the least data volume and containing multiple target video frames, avoid the existence of too many useless video frames in the target segment, and quickly decode to obtain the image sequence of the alarm object.

[0122] Of course, without considering the data amount, any I frame in the specified code stream segment whose timestamp is smaller than the start timestamp can be determined as the target start timestamp, which is reasonable.

[0123] Adaptively, decoding the plurality of target video frames from the target segment to obtain an image sequence corresponding to the alarm object includes:

[0124] Reading and decoding the target segment from the target buffer;

[0125] A plurality of target video frames having the frame identification information are selected from the decoded video frames to obtain an image sequence corresponding to the alarm object.

[0126] After determining the target segment, the target segment can be read and decoded from the target buffer area to obtain the video frames corresponding to the target segment; the decoded video frames are continuous video frames from the I frame of the target start timestamp to the last frame of multiple target video frames. The purpose of this application is to enable the second analysis model to use more image data to perform alarm analysis on the alarm object, and multiple target video frames contain the alarm object of the alarm object. Therefore, multiple target video frames with frame identification information can be selected from the decoded video frames to obtain an image sequence corresponding to the alarm object. The image sequence is more image data used by the second analysis model for secondary analysis of the alarm object.

[0127] A data processing method provided by an embodiment of the present application can determine the initial timestamp range according to the frame identification information of multiple target video frames, and identify the target segment in the specified code stream segment according to the initial timestamp range; and the target buffer area can cache the target code stream and the local timestamp of each I frame in the target code stream, and can determine the target start timestamp according to the end timestamp, so that the segment in the code stream segment that is within the local timestamp range formed by the end timestamp and the target start timestamp can be specified as the target segment with less data volume. In addition, the embodiment of the present application can directly reuse the target buffer area without additional memory allocation, and can directly read and decode the target segment from the target buffer area, and select multiple target video frames with frame identification information from the decoded video frames to obtain the image sequence corresponding to the alarm object.

[0128] A data processing method provided in an embodiment of the present application is introduced below by way of comparison.

[0129] The secondary analysis process of the second analysis model (large model) of the existing solution is as follows:

[0130] The front-end device reports the alarm image obtained after analyzing the target code stream to the back-end device. The back-end device calls the large model for secondary analysis based on the alarm image, thereby generating an alarm. The alarm at this time is the alarm generated by the front-end device after the large model performs secondary analysis on the alarm image and filters it.

[0131] This application is applicable to different object recognition methods and can be widely used in various deep learning algorithm applications. The method of filtering the image sequence of video frames from the code stream is through alarm ID, object recognition box, video frame relative time, absolute time and other common object description information. At present, the front-end devices that support object recognition can superimpose the object description information of each object in the code stream. Figure 2 As shown, the secondary analysis process of the second analysis model (large model) of this application is as follows:

[0132] The front-end device reports the alarm image after analyzing the target code stream to the back-end device, and reports the target code stream and the object description information of each object identified by the alarm analysis of the target code stream to the back-end device at the same time; the back-end device decodes the target code stream to obtain an image sequence of multiple target video frames based on the alarm description information of the alarm object in the alarm image and the description information of each object; then, based on the image sequence of multiple target video frames and the alarm image, the large model is called for secondary analysis to generate an alarm. At this time, the alarm is an alarm generated by the front-end device after the large model performs secondary analysis on the image sequence of multiple target video frames and the alarm image, and then filters the alarm. Since the alarm analysis is performed through more image data (i.e., the image sequence of multiple target video frames and the alarm image), the underreporting of the alarm analysis can be reduced.

[0133] Of course, in the solution of the present application, it is also possible to call a large model for secondary analysis based on the alarm image to generate an alarm, which is reasonable.

[0134] In another implementation, Figure 6 As shown, the present application can also first identify whether the frame selection is successful. If the recognition result is yes, that is, the image sequence corresponding to the alarm object is selected, and the image sequence can be sent for secondary analysis (that is, the image sequence is subjected to secondary analysis through the second analysis model); if the recognition result is no, that is, the image sequence corresponding to the alarm object is not selected (at this time, the target start timestamp or the start timestamp of the initial timestamp range may be no later than the start time of the specified code stream segment, resulting in the inability to obtain the full image sequence), at this time, the alarm image obtained by the front-end first analysis model can be sent for secondary analysis (that is, the alarm image is subjected to secondary analysis through the second analysis model).

[0135] It can be seen that in the present application, multiple target video frames contain alarm objects, and the time information of the multiple target video frames is no later than the time information corresponding to the alarm video frame. The image sequence of the multiple target video frames can represent more object information of the alarm object before the alarm. The image sequence can be processed for alarm analysis through the second analysis model, so as to accurately retain or eliminate the alarm object to reduce underreporting of the alarm analysis.

[0136] A data processing method provided by the present application is introduced below in conjunction with another embodiment.

[0137] Taking the front-end device as the front-end IPC as an example, the process of the first analysis of the front-end IPC is omitted here. The process of a data processing method provided in an embodiment of the present application is as follows Figure 3As shown, the front-end IPC reports stream 1, alarm pictures, alarm description information, etc. to the back-end device; wherein, stream 1 is the above-mentioned stream data, including the target stream and the intermediate analysis result; the back-end device decapsulates stream 1 to obtain stream 2 (target stream), object ID, identification box, etc. (intermediate analysis result), based on stream 2, object ID, identification box, and alarm ID+object type (alarm type of alarm object)+identification box (identification box of alarm object) in the alarm description information, selects an image sequence of multiple target video frames from stream 2 based on the frame selection scheme, and sends the image sequence to the large model for secondary analysis to obtain the target analysis result of retaining or eliminating the alarm object; after each alarm object is analyzed through the above process, the remaining retained alarm objects are processed for alarm (during alarm processing, the generated alarm can be associated with the corresponding alarm picture, alarm description information, etc.).

[0138] Among them, the front-end device will package the intermediate analysis results of the first analysis in the form of private information when encapsulating the target code stream ps (PDU Specific, protocol data unit details); at the same time, the front-end device will report the alarm results of the first analysis (alarm picture, alarm description information, etc.) to the back-end device.

[0139] In this application, the front-end IPC can superimpose the intermediate analysis results in the target code stream, and report the initial analysis results (alarm pictures, alarm description information, etc.) to the back-end device. The back-end device selects an image sequence from the target code stream based on the frame selection scheme and the received analysis results; the image sequence of multiple target video frames can represent more object information of the alarm object before the alarm, and the image sequence can be processed for alarm analysis through a large model, so as to accurately retain or eliminate the alarm object to reduce underreporting of the alarm analysis.

[0140] The following is a detailed introduction to a method for selecting an image sequence in a data processing method provided by the present application in conjunction with a specific embodiment.

[0141] like Figure 4 As shown, the method for selecting the image sequence in this application includes the following steps:

[0142] S401: decapsulate the bitstream data; that is, after receiving the bitstream data, the backend device first decapsulates the bitstream data to obtain the target bitstream and IVS information, where the IVS information is the private frame information of ps (the bitstream data is encapsulated according to the ps protocol); wherein the IVS information is the object description information of each object identified when the front-end first analysis model performs alarm analysis on the target bitstream, that is, the above-mentioned intermediate analysis result.

[0143] S402: caching the target code stream into the target buffer area; that is, after decapsulating the code stream data, the obtained target code stream can be cached into the target buffer area, so as to subsequently decode and obtain an image sequence of multiple target video frames.

[0144] S403: Cache the IVS information to the target buffer area; that is, after decapsulating the bitstream data, the obtained IVS information can be cached to the target buffer area, so that the IVS information can be used to determine multiple target video frames later.

[0145] The IVS information includes the ID of the identified object, the identification frame of the object, the backend Timestamp frame time (timestamp), the backend absolute time, the backend relative time, and the like.

[0146] S404: Preliminarily filter the target bitstream based on the front-end alarm time to obtain a designated bitstream segment; after the target bitstream is cached, in order to facilitate subsequent analysis, the bitstream can be preliminarily filtered based on the front-end alarm time (the alarm time is the local absolute time of the back-end device after the back-end device converts the front-end alarm time) to obtain a designated bitstream segment; wherein, the interception duration of the designated bitstream segment can be determined based on factors such as the size of the buffer area and the level of decoding performance. Due to the influence of the size of the buffer area and the level of decoding performance, the interception duration can be the duration of any bitstream segment containing multiple target video frames. Exemplarily, the bitstream 3s before the alarm time in the target bitstream can be used as the designated bitstream segment.

[0147] S405: Preliminarily filter the IVS information according to the front-end alarm time to obtain description information of each first object; after the IVS information is cached, in order to facilitate subsequent analysis, the IVS information can be preliminarily filtered according to the front-end alarm time and the front-end absolute time of the IVS information to obtain description information of each first object; illustratively, the information in the IVS information 5s before the alarm time can be used as description information of each first object.

[0148] S406: Selecting second object description information from each first object description information; after initially screening each first object description information, for any alarm object in the front-end alarm information, selecting object description information that is the same object as the alarm object from each first object description information to obtain second object description information. The second object description information may indicate multiple time information corresponding to the object, so that frame identification information of multiple target video frames can be determined.

[0149] S407: According to the frame identification information, a target segment is determined from the specified bitstream segment, and an image sequence of multiple target video frames is decoded from the target segment; after determining the frame identification information, the target segment can be determined from the specified bitstream segment according to the frame identification information, and an image sequence of multiple target video frames can be decoded from the target segment, and the obtained image sequence of multiple target video frames can be subsequently analyzed twice by a second analysis model.

[0150] In the embodiment of the present application, the back-end device can decapsulate the bitstream data, cache the obtained target bitstream and IVS information, and preliminarily screen the target bitstream and IVS information according to the front-end alarm time to reduce the amount of data analyzed; and for any alarm object in the front-end alarm information, select the second object description information from each first object description information, and determine the frame identification information, so as to determine the target segment from the preliminarily screened specified bitstream segment according to the frame identification information, and decode the image sequence of multiple target video frames. It can be seen that when selecting and obtaining the image sequence, the present application can preliminarily screen the target bitstream and IVS information, etc. to reduce the amount of data analyzed, so that the image sequence of multiple target video frames can be quickly obtained.

[0151] At present, the video stream analysis is generally performed by the upper layer application searching the video from the video file, intercepting a fixed length of time, such as the video file 5 seconds before the alarm time, and then sending it for analysis. This solution requires searching the file from the video file, and the efficiency of reading the file through IO is slow. In addition, if multiple channels are opened for analysis, if pre-reading is performed, a larger memory space needs to be opened for each additional channel of analysis, which consumes a lot of memory resources.

[0152] In the present application, the target cache area of ​​the cache code stream can be directly reused, and at the same time, the I frame information can be saved so as to determine the target segment with the least amount of data and containing multiple target video frames, avoid too many useless video frames in the target segment, and quickly decode to obtain the image sequence of the alarm object.

[0153] The process of saving I frame information is as follows Figure 5a As shown, when the target code stream is encapsulated and cached in the target buffer area, for each video frame in the target code stream, it is identified whether the video frame is an I frame. If yes, the I frame timestamp and the position in the target buffer area are recorded; if not, the next video frame is selected, and the identification of whether the video frame is an I frame continues until all video frames are identified.

[0154] For each I frame, the recorded information such as the address of the I frame in the buffer area, the corresponding timestamp, etc. are all cached in the I frame information buffer area of ​​the target buffer area.

[0155] Afterwards, the recorded I-frame information can be used to determine the target segment with the least amount of data and containing multiple target video frames. The process of selecting the target segment is as follows: Figure 5b As shown, after obtaining the frame identification information, the initial timestamp range is first determined according to the frame identification information. At this time, the initial timestamp range is not the timestamp range with the least amount of data. Therefore, the end timestamp in the initial timestamp range can be determined, and based on the position of the end timestamp, an I frame is found forward. In order to include multiple target video frames, it is continued to be determined whether the I frame timestamp is less than the start timestamp. If so, the I frame is determined as the target start timestamp, and the fragment between the target start timestamp and the end timestamp is determined as the target fragment. The target fragment is copied from the target buffer area for decoding; if not, return to the step of finding an I frame forward.

[0156] It can be seen that the present application can reuse the target buffer area, can cache the target code stream and the timestamp, position and other information of the I frame, and, through the saved I frame information, can determine the target segment with the least amount of data and containing multiple target video frames, avoid too many useless video frames in the target segment, so as to quickly decode and obtain the image sequence of the alarm object. When copying the target segment, the present application can directly read the target segment from the target buffer area, without having to open up additional memory space, which can save memory resources.

[0157] The following is an introduction to the steps of calling the second analysis model to perform alarm analysis on the image sequence in the present application in conjunction with another embodiment.

[0158] like Figure 7 As shown, calling the second analysis model to perform alarm analysis processing on the image sequence may include the following steps:

[0159] S701: Divide the image sequence into three image groups; that is, divide the image sequence of multiple target video frames into three image groups; in this embodiment, the image sequence can be 24 images, and the 24 images can be divided into three groups in chronological order. Of course, the number of images in the image sequence and the number of divided image groups are merely examples and should not constitute a limitation on the present application.

[0160] S702: The image group closest to the alarm time is taken as the current image group, and is input into the second analysis model for secondary analysis; after the division is completed, the image group closest to the alarm time is taken as the current image group, and the current image group is input into the second analysis model for secondary analysis to obtain the secondary analysis result.

[0161] S703: Identify inconsistencies between the front-end analysis results and the secondary analysis results?

[0162] That is, after obtaining the secondary analysis result, identify whether the front-end analysis result and the secondary analysis result are inconsistent, for example: identify whether the front-end analysis result is a person or a car, and the secondary analysis result is other? If the recognition result is yes, that is, the front-end and back-end results are inconsistent, then execute step S705; if the recognition result is no, that is, the front-end and back-end results are consistent, then execute step S704 to obtain the target analysis result retaining the alarm object.

[0163] S704: Obtain the target analysis result; that is, if the results of the front end and the back end are consistent, the target analysis result that retains the alarm object is obtained; or, if the results of the front end and the back end are inconsistent, it is recognized that there is no previous image group, and the target analysis result that eliminates the alarm object is obtained.

[0164] S705: Is it recognized that there is no previous image group?

[0165] That is, when the results of the front end and the back end are inconsistent, it is identified whether the current image group does not have a previous image group. If the identification result is yes, that is, the current image group does not have a previous image group, execute step S704 to obtain the target analysis result of eliminating the alarm object; if the identification result is no, that is, the current image group has a previous image group, execute step S706.

[0166] S706: Considering that there is occlusion, taking one image group forward as a new current image group, and inputting it into the second analysis model for secondary analysis;

[0167] That is, if there is a previous image group in the current image group, it is considered that the alarm object of the alarm object is blocked, and one image group is taken as the new current image group and input into the second analysis model for secondary analysis.

[0168] It can be seen that in the embodiment of the present application, the image sequence can be divided into multiple image groups, and the most recent current image group can be selected according to the alarm time and input into the second analysis model for alarm analysis, thereby reducing the amount of image data analyzed by the second analysis model. In addition, if the secondary analysis result of the current image group is inconsistent with the front-end analysis result, a new current image group can be continuously selected and input into the second analysis model for secondary analysis. This can reduce the amount of data for a single analysis by the second analysis model while ensuring that the second analysis model uses more image data to analyze the alarm object.

[0169] The following is a detailed introduction to the process of determining the front-end relative time in the present application in conjunction with a specific embodiment.

[0170] like Figure 8As shown, in the code stream data: the I frame information contains absolute time T1 and relative time t1; the IVS information contains relative time t2; the I frame absolute time: fe_T1, the I frame relative time: fe_t1; the P frame relative time fe_t2, the relative time fe_t2 in the IVS information corresponding to the P frame.

[0171] After decapsulation, in the target code stream (back-end ps stream), the absolute time of the I frame is: be_T1, the relative time of the I frame is: be_t1; the relative time of the P frame is be_t2, and the front-end absolute time fe_T2 of the IVS information is calculated to be fe_T1+ be_t2-be_t1 (when the front-end device encapsulates the code stream data, the front-end relative time fe_t1 and fe_t2 may be lost, be_t2- be_t1= fe_t2- fe_t1); the relative time be_t2 is converted by fe_t2 based on the mapping relationship between the front-end relative time and the back-end relative time (that is, the conversion relationship between the front-end device and the back-end device about the relative time); illustratively, it can be obtained based on the time difference between the front-end sending the code stream data and the back-end receiving the target code stream after decapsulation, and fe_t2.

[0172] The front-end absolute time fe_T2 of the IVS information is fe_T1+ be_t2- be_t1; the relative time be_t2 is converted from fe_t2.

[0173] It can be seen that in an embodiment of the present application, before selecting each first object description information from the object description information of each object, the front-end absolute time corresponding to the relative time of each object can also be determined, so as to accurately select each first object description information based on the front-end absolute time of the alarm object and the front-end absolute time of each object.

[0174] The present application also provides an electronic device, such as Fig. 9 As shown, including:

[0175] Memory 901, used for storing computer programs;

[0176] The processor 902 is used to implement any of the above-mentioned data processing methods when executing the program stored in the memory 901.

[0177] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 902, the communication interface, and the memory 901 communicate with each other via the communication bus.

[0178] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0179] The communication interface is used for communication between the above electronic device and other devices.

[0180] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0181] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0182] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data processing methods are implemented.

[0183] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the data processing methods in the above embodiments.

[0184] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0185] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0186] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0187] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: Applied to an electronic device, the electronic device being a backend device for calling a second analysis model, the method comprising: Receive the bitstream data and initial analysis results reported by the front-end device; wherein the bitstream data includes data obtained by encapsulating the target bitstream and the intermediate analysis results by the front-end device; the intermediate analysis results include object description information of each object identified by the front-end device calling the first analysis model to perform alarm analysis on the target bitstream; the initial analysis results include alarm description information of the alarm object obtained by the alarm analysis; Based on the alarm description information and the object description information of each object, frame identification information of multiple target video frames belonging to the target code stream is determined; wherein the multiple target video frames contain the alarm objects indicated by the alarm description information, and time information corresponding to the multiple target video frames is no later than time information corresponding to the alarm video frames indicated by the alarm description information; Based on the determined frame identification information, the plurality of target video frames are decoded from the target bitstream to obtain an image sequence corresponding to the alarm object; The second analysis model is called to perform alarm analysis processing on the image sequence to obtain a target analysis result corresponding to the alarm object; wherein the target analysis result is used to indicate whether to retain or eliminate the alarm object.

2. The method according to claim 1, characterized in that Before decoding the plurality of target video frames from the target bitstream based on the determined frame identification information to obtain the image sequence corresponding to the alarm object, the method further includes: Determine a designated code stream segment in the target code stream; wherein the designated code stream segment is a code stream segment having a first duration and having an end time represented by the alarm time information in the alarm description information; The step of decoding the plurality of target video frames from the target bitstream based on the determined frame identification information to obtain an image sequence corresponding to the alarm object includes: Determine a target segment in the designated bitstream segment that includes the video frame indicated by the frame identification information; The multiple target video frames are decoded from the target segment to obtain an image sequence corresponding to the alarm object.

3. The method according to claim 1 or 2, characterized in that: The step of determining frame identification information of a plurality of target video frames belonging to the target code stream based on the alarm description information and the object description information of each object includes: Based on the alarm time information in the alarm description information, each first object description information is selected from the object description information of each object; wherein, in each first object description information, the time represented by the time information about the object is within a time range having a second duration and ending at the time represented by the alarm time information; the time information of any object is the time information of the video frame in which the object is identified; From each of the first object description information, select the object description information in which the indicated object and the alarm object in the alarm description information are the same object, to obtain the second object description information; Based on the time information corresponding to the object in the obtained second object description information, frame identification information of multiple target video frames belonging to the target code stream is determined.

4. The method according to claim 3, characterized in that The alarm time information in the alarm description information is: the front-end absolute time when the front-end device recognizes the alarm object; the time information about the object in the object description information of each object includes: the relative time of the video frame at which the object is recognized; Before selecting each first object description information from the object description information of each object based on the alarm time information in the alarm description information, the method further includes: Determine the front-end absolute time corresponding to the relative time of each object; The selecting each first object description information from the object description information of each object based on the alarm time information in the alarm description information includes: Based on the front-end absolute time of the alarm object and the front-end absolute time of each object, each first object description information is selected from the object description information of each object.

5. The method according to claim 2, characterized in that: The determining of the target segment in the designated code stream segment that includes the video frame indicated by the frame identification information includes: Determine an initial timestamp range according to the frame identification information of the multiple target video frames; wherein the starting timestamp of the initial timestamp range is the local timestamp of the first frame of the multiple target video frames, and the ending timestamp is the local timestamp of the last frame of the multiple target video frames; the local timestamp of any frame is the absolute time with respect to the local clock of the electronic device; According to the initial timestamp range, a target segment in the designated code stream segment that includes the video frame indicated by the frame identification information is identified.

6. The method according to claim 5, characterized in that The method further comprises: In response to receiving the bitstream data, buffering a target bitstream in the bitstream data into a target buffer area, and recording a local timestamp of an I frame of the target bitstream; The step of identifying, according to the initial timestamp range, a target segment in the designated bitstream segment that includes the video frame indicated by the frame identification information comprises: Determine a target start timestamp according to an end timestamp in the initial timestamp range and a local timestamp of each I frame of the specified code stream segment; wherein the target start timestamp is a timestamp of an I frame, and the target start timestamp is smaller than a start timestamp in the initial timestamp range; A segment of the designated code stream segment that is within a local timestamp range formed by the end timestamp and the target start timestamp is determined as a target segment.

7. The method according to claim 6, characterized in that The determining the target start timestamp according to the end timestamp within the initial timestamp range and the local timestamp of each I frame of the designated code stream segment includes: Determine the local timestamp of the first I frame in the specified code stream segment that is located before the end timestamp of the initial timestamp range as the current timestamp; Identify the magnitude relationship between the current timestamp and the starting timestamp within the initial timestamp range; In response to the current timestamp being less than the starting timestamp within the initial timestamp range, taking the current timestamp as the target starting timestamp; In response to the current timestamp being not less than the starting timestamp within the initial timestamp range, the timestamp of the previous I frame of the current timestamp is used as the new current timestamp, and the step of identifying the size relationship between the current timestamp and the starting timestamp within the initial timestamp range is returned.

8. The method according to claim 6, characterized in that Decoding the plurality of target video frames from the target segment to obtain an image sequence corresponding to the alarm object includes: Reading and decoding the target segment from the target buffer; A plurality of target video frames having the frame identification information are selected from the decoded video frames to obtain an image sequence corresponding to the alarm object.

9. The method according to claim 1, characterized in that: The calling of the second analysis model to perform alarm analysis processing on the image sequence to obtain a target analysis result corresponding to the alarm object includes: Dividing the image sequence into a plurality of image groups according to time sequence; A target image group among the multiple image groups is used as the current image group; wherein the time represented by the time information of the video frames in the target image group is closest to the time represented by the alarm time information in the alarm description information relative to other image groups; Inputting the current image group into the second analysis model, so that the second analysis model performs alarm analysis on the current image group to obtain a secondary analysis result corresponding to the current image group; In response to the secondary analysis result corresponding to the current image group indicating the existence of the alarm object, retaining the alarm object as the target analysis result of the alarm object; In response to the secondary analysis result corresponding to the current image group indicating that the alarm object does not exist, if there is a previous image group for the current image group, the previous image group is used as the current image group, and the step of inputting the current image group into the second analysis model is returned so that the second analysis model performs alarm analysis on the current image group to obtain the secondary analysis result corresponding to the current image group; if there is no previous image group for the current image group, the alarm object is eliminated as the target analysis result of the alarm object.

10. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-9 when executing a program stored in a memory.

Images