A video-driven multi-component synchronized display method and apparatus

Abstract

The application discloses a kind of video-driven multi-component synchronous display method and device, it is related to video display technical field, main purpose is to realize video picture, each sensor data and ship state information synchronous display.The main technical scheme of the present application is: according to the creation time and end time corresponding to the video data to be played back, the sensor data and ship state data stored in the database are traversed, the target sensor data and target ship state data corresponding to the target time period are obtained;Based on the time corresponding to each frame of video data, the target sensor data and target ship state data corresponding to each frame of video data are determined using a predetermined corresponding rule;Display the video data to be played back, display the corresponding target sensor data, and draw a ship state graph.The present application is used for offshore oil cleaning scene video display.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video display, in particular to a video-driven multi-component synchronous display method and device. BACKGROUND

[0002] OceanEye oil spill search system is a hardware developed by Maritime Robotics Company based on a shipborne floating air balloon platform to search for oil spills on the sea surface. The supporting control software can realize the functions of operating the whole device, displaying the values of various sensors, playing real-time video stream of the floating platform, taking screenshots (photos) and recording pictures on demand, etc. It is widely used in offshore oil cleaning operations to provide information about oil spills on the sea surface.

[0003] However, the supporting control software of OceanEye oil spill search system only displays real-time data and records video stream. When playing back the video, it cannot feedback the values of various sensors and the position information of the ship at the current picture, so it cannot intuitively reproduce the on-site environment when accident backtracking and experience analysis and summary are performed due to the lack of comprehensive data parameters. SUMMARY

[0004] In view of the above problems, the present application provides a video-driven multi-component synchronous display method and device, the main purpose of which is to realize the synchronous display of video pictures, various sensor data and ship state information, and provide more comprehensive information parameters.

[0005] To solve the above technical problems, the present application proposes the following solutions:

[0006] In a first aspect, the present application provides a video-driven multi-component synchronous display method, which is applied to an offshore oil cleaning scene, comprising:

[0007] obtaining a target time period corresponding to the video data to be played back, the target time period being determined according to a creation time and an end time;

[0008] traversing the stored sensor data and ship state data in the database according to the creation time and the end time to obtain target sensor data and target ship state data corresponding to the target time period, wherein the sensor data and ship state data have the same format and time stamp;

[0009] determining the target sensor data and target ship state data corresponding to each frame of video data of the video data to be played back based on the time corresponding to each frame of video data using a preset corresponding rule in the target sensor data and target ship state data corresponding to the target time period;

[0010] displaying the to-be-played video data, displaying the target sensor data corresponding to each frame of the to-be-played video data, and drawing a ship state graph based on the target ship state data corresponding to each frame of the to-be-played video data.

[0011] Preferably, the target sensor data and the target ship state data corresponding to each frame of the to-be-played video data are determined in the target sensor data and the target ship state data corresponding to the target time period by using a preset correspondence rule based on the time corresponding to each frame of the to-be-played video data, and the method comprises the following steps of:

[0012] arranging the target sensor data and the target ship state data in ascending order of time to generate a target data linked list;

[0013] According to the target data linked list, the time corresponding to each frame of the to-be-played video data is matched by using a preset correspondence rule to obtain the target sensor data and the target ship state data corresponding to each frame of the to-be-played video data.

[0014] Preferably, the target sensor data and the target ship state data corresponding to each frame of the to-be-played video data are determined in the target sensor data and the target ship state data corresponding to the target time period by using a preset correspondence rule based on the time corresponding to each frame of the to-be-played video data, and the method comprises the following steps of:

[0015] obtaining the start playing time corresponding to each frame of the to-be-played video data;

[0016] Based on the start playing time corresponding to each frame of the to-be-played video data, the target sensor data and the target ship state data corresponding to each frame of the to-be-played video data are obtained by traversing the target data linked list.

[0017] Preferably, the target sensor data and the target ship state data corresponding to each frame of the to-be-played video data are determined in the target sensor data and the target ship state data corresponding to the target time period by using a preset correspondence rule based on the time corresponding to each frame of the to-be-played video data, and the method comprises the following steps of:

[0018] obtaining a target sensor data queue and a target ship state data queue of the target data linked list, wherein the target sensor data in the target sensor data queue and the target ship state data in the target ship state data queue correspond to each other in ascending order of time, and the target sensor data and the target ship state data corresponding to each other have the same time stamp;

[0019] The target sensor data and the target ship state data corresponding to each other are grouped into a target data group to obtain a target data group queue, and each target data group has a unique start time.

[0020] traversing the target data group queue according to the start playing time corresponding to each frame of video data, matching the target data group in which the start playing time corresponding to each frame of video data is located, and obtaining the start time of the target data group;

[0021] determining whether the time difference between the start playing time corresponding to each frame of video data and the start time of the target data group is not less than a preset threshold value;

[0022] If not, determining that the target data group in which the start playing time corresponding to each frame of video data is located is the target data group corresponding to each frame of video data;

[0023] If yes, determining the next target data group of the target data group in which the start playing time corresponding to each frame of video data is located as the target data group corresponding to each frame of video data in ascending order of time;

[0024] obtaining the target sensor data and the target ship state data corresponding to each frame of video data based on the target data group.

[0025] Preferably, before the target time period corresponding to the video data to be played back is obtained, the method comprises:

[0026] obtaining sensor data, ship state data and video data of the same time period;

[0027] unifying the format of the sensor data and the ship state data at the same time according to a preset time interval, and marking the timestamp to obtain processed sensor data and ship state data;

[0028] storing the processed sensor data and ship state data to a database to obtain stored sensor data and ship state data.

[0029] Preferably, the sensor data, ship state data and video data of the same time period are obtained by:

[0030] obtaining data messages containing sensor data of a preset time period;

[0031] parsing the data messages into plaintext data by using a preset protocol to obtain the sensor data;

[0032] obtaining the ship state data of the preset time period;

[0033] recording video corresponding to the preset time period according to a preset parameter to obtain the video data, wherein the preset parameter at least includes resolution, compression mode and frame rate.

[0034] Preferably, the target time period corresponding to the video data to be played back is obtained, the target time period is determined according to a creation time and an end time, and the target time period comprises:

[0035] The creation time and the duration of the video data to be played back are obtained based on the stored video data to be played back.

[0036] The end time corresponding to the creation time and the duration of the video data to be played back is calculated.

[0037] The target time period corresponding to the creation time and the end time is calculated.

[0038] In a second aspect, the present application provides a monitoring device for object state, which is applied to offshore oil cleaning scene, and comprises:

[0039] A first obtaining unit is configured to obtain a target time period corresponding to video data to be played back, the target time period being determined according to a creation time and an end time.

[0040] A second obtaining unit is configured to traverse sensor data and ship state data stored in a database according to the creation time and the end time, to obtain target sensor data and target ship state data corresponding to the target time period, wherein the sensor data and the ship state data have the same format and time stamp.

[0041] A determining unit is configured to determine, based on a preset corresponding rule, target sensor data and target ship state data corresponding to each frame of video data of the video data to be played back in the target sensor data and the target ship state data corresponding to the target time period, according to a time corresponding to each frame of video data of the video data to be played back.

[0042] A display unit is configured to display the video data to be played back, display the target sensor data corresponding to each frame of video data of the video data to be played back, and draw a ship state graph based on the target ship state data corresponding to each frame of video data of the video data to be played back.

[0043] Preferably, the determining unit comprises:

[0044] A generating module is configured to generate a target data linked list by arranging the target sensor data and the target ship state data in ascending order of time.

[0045] A matching module is configured to match, according to the target data linked list, a time corresponding to each frame of video data of the video data to be played back by using a preset corresponding rule, to obtain target sensor data and target ship state data corresponding to each frame of video data of the video data to be played back.

[0046] Preferably, the matching module comprises:

[0047] Further used for acquiring a starting playing time corresponding to each frame of video data of the video data to be played back;

[0048] Further used for traversing the target data linked list based on the starting playing time corresponding to each frame of video data, to obtain target sensor data and target ship state data corresponding to each frame of video data.

[0049] Preferably, the matching module comprises:

[0050] A first acquisition submodule is used for acquiring a target sensor data queue and a target ship state data queue of the target data linked list, wherein the target sensor data in the target sensor data queue and the target ship state data in the target ship state data queue correspond to each other in ascending order of time, and the target sensor data and the target ship state data corresponding to each other have the same time stamp;

[0051] A composition submodule is used for composing the target sensor data and the target ship state data corresponding to each other into a target data group, to obtain a target data group queue, and each target data group has a unique starting time;

[0052] A matching submodule is used for traversing the target data group queue according to the starting playing time corresponding to each frame of video data, matching the target data group in which the starting playing time corresponding to each frame of video data is located, and acquiring the starting time of the target data group;

[0053] A judgment submodule is used for judging whether a time difference between the starting playing time corresponding to each frame of video data and the starting time of the target data group is not less than a preset threshold value;

[0054] A determination submodule is used for determining, if the time difference between the starting playing time corresponding to each frame of video data and the starting time of the target data group is less than the preset threshold value, that the target data group in which the starting playing time corresponding to each frame of video data is located is the target data group corresponding to each frame of video data;

[0055] The determination submodule is further used for determining, if the time difference between the starting playing time corresponding to each frame of video data and the starting time of the target data group is not less than the preset threshold value, that a next target data group of the target data group in which the starting playing time corresponding to each frame of video data is located in ascending order of time is the target data group corresponding to each frame of video data;

[0056] A second acquisition submodule is used for obtaining the target sensor data and the target ship state data corresponding to each frame of video data based on the target data group.

[0057] Preferably, the device comprises:

[0058] a third acquisition unit, configured to acquire sensor data, ship state data and video data of a same time period;

[0059] a processing unit, configured to uniformly format the sensor data and the ship state data of the same time according to a preset time interval, and mark the time stamp, to obtain processed sensor data and ship state data;

[0060] a storage unit, configured to store the processed sensor data and ship state data to a database, to obtain stored sensor data and ship state data.

[0061] Preferably, the third acquisition unit comprises:

[0062] a first acquisition module, configured to acquire data messages containing sensor data of a preset time period;

[0063] a parsing module, configured to parse the data messages into plaintext data by using a preset protocol, to obtain the sensor data;

[0064] a second acquisition module, configured to acquire the ship state data of the preset time period;

[0065] a third acquisition module, configured to record a video corresponding to the preset time period according to preset parameters, to obtain the video data, wherein the preset parameters at least include resolution, compression mode and frame rate.

[0066] Preferably, the first acquisition unit comprises:

[0067] an acquisition module, configured to acquire a creation time and a duration of the to-be-played-back video data based on stored to-be-played-back video data;

[0068] a first calculation unit, configured to calculate a corresponding end time according to the creation time and the duration of the to-be-played-back video data;

[0069] a second calculation unit, configured to calculate a corresponding target time period according to the creation time and the end time.

[0070] In order to achieve the above-mentioned purposes, according to a third aspect of the present application, a storage medium is provided, the storage medium comprises a stored program, wherein when the program runs, the device where the storage medium is located is controlled to execute the video-driven multi-component synchronous display method of the video-driven multi-component synchronous display method of the first aspect.

[0071] In order to achieve the above object, according to a fourth aspect of the present application, there is provided an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing all or part of the steps of the multi-component synchronous display device for video-driven multi-component synchronous display of the second aspect when executing the program.

[0072] By means of the above technical solution, the method and device for video-driven multi-component synchronous display provided by the present application can record the sensor information, ship information and video state information and synchronously display the video picture, each sensor data and ship state information, thereby providing more comprehensive information parameters for the oil cleaning work at sea.

[0073] The above description is only a summary of the technical solution of the present application. In order to more clearly understand the technical means of the present application, the content of the specification can be implemented, and in order to make the above and other objects, features and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0074] Various other advantages and benefits will become apparent to those of ordinary skill in the art, upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to depict only preferred embodiments of the application, and therefore should not be considered to narrow the scope of the present application in any way. Similarly, it is intended that the specification and drawings serve as an exemplification of the application, and should not be considered as limiting its broadest scope. Moreover, as used herein, the term "or" is intended to mean any one of the items, a combination of any of the items, or all of the items. In other words, the term "or" is used in the inclusive sense, not the exclusive sense. In addition, the term "comprising" is used throughout the specification and in claims to mean that the features, steps, or components so described can be present or combined, but are not limited to, the described features, steps, or components. Furthermore, the use of the term "about" is intended to mean approximately, nearly, almost, virtually, at about, at or about, at or circa, at the region of, more or less, nearly the same as, most

[0075] Figure 1 A flow chart of a method for video-driven multi-component synchronous display is shown according to an embodiment of the present application;

[0076] Figure 2 Another video-driven multi-component synchronous display method provided by the embodiment of the present application is shown in a flow chart.

[0077] Figure 3 A component block diagram of a video-driven multi-component synchronous display device provided by the embodiment of the present application is shown.

[0078] Figure 4 Another component block diagram of a video-driven multi-component synchronous display device provided by the embodiment of the present application is shown.

[0079] Figure 5 An example diagram of another video-driven multi-component synchronous display method provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0080] Exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0081] With the development of modern cameras, computers and network technology, people have strong demand for video and related data synchronous display in special equipment. However, there are few realizations in the field of special equipment for offshore oil cleaning operations. OceanEye oil spill searching system is a hardware developed by Maritime Robotics Company based on a shipborne floating air balloon platform to search for oil spills on the sea. Its supporting control software can realize the functions of operating the overall device, displaying the values of each sensor, playing real-time video stream of the floating platform, taking screenshots (photos) and recording pictures on demand, etc. However, in terms of video review function, only the player provided by the operating system is used to play video information, and the sensor information cannot be played back.

[0082] To solve this problem, the inventor thought of recording the sensor information, ship information and video state information and synchronously displaying the video pictures, sensor data and ship state information.

[0083] Therefore, the embodiment of the present application provides a video-driven multi-component synchronous display method, which synchronously displays the video pictures, sensor data and ship state information. The specific execution steps are as shown in Figure 1 The method comprises the following steps.

[0084] 101, obtaining a target time period corresponding to the video data to be played back, the target time period being determined according to the creation time and the end time.

[0085] In the oil cleaning operation on the sea, the picture on the sea surface needs to be recorded first, and the recorded video also needs to be stored for subsequent video playback. The video is recorded and stored according to a certain time period, so each piece of stored video has a creation time and an end time.

[0086] The video data to be played back is a video stream recorded and stored by a video playing device for subsequent playback. The target time period is determined by the creation time and the end time, for example, the creation time of A video is 8 o'clock and the end time is 9 o'clock, so the target time period is 8 o'clock-9 o'clock.

[0087] 102. According to the creation time and the end time, the stored sensor data and ship state data in the database are traversed to obtain the target sensor data and the target ship state data corresponding to the target time period.

[0088] Among them, the sensor data and the ship state data have the same format and timestamp, that is, each piece of recorded sensor data and ship state data has the same start recording time and end recording time.

[0089] In the oil cleaning operation on the sea, the sensor data and the ship state data are recorded into the database at the same time when the video is recorded; according to the creation time and the end time of the video data to be played back, the start recording time and the end recording time of each piece of stored sensor data and ship state data in the database are traversed, the sensor data and the ship state data recorded between the creation time and the end time of the video data to be played back can be matched, that is, the target sensor data and the target ship state data corresponding to the target time period are obtained.

[0090] 103. Based on the time corresponding to each frame of video data of the video data to be played back, the target sensor data and the target ship state data corresponding to each frame of video data are determined in the target sensor data and the target ship state data corresponding to the target time period according to a preset corresponding rule.

[0091] According to the prior art, the recording start time and the recording end time of each frame of video data of the video data to be played back can be obtained by frame rate, and the specific manner is not limited in this step. Therefore, according to the frame rate, the time corresponding to each frame of video data of the video data to be played back is obtained, which includes the recording start time and the recording end time.

[0092] The preset corresponding rule is the correspondence between different kinds of data according to the same recording time.

[0093] According to step 102, the target sensor data and the target ship state data corresponding to the target time period can be obtained, and each frame of video data of the to-be-played-back video data is matched with the target sensor data and the target ship state data corresponding to the target time period based on the same recording time in chronological order, so that the target sensor data and the target ship state data corresponding to each frame of video data when the to-be-played-back video is played can be determined.

[0094] 104, display the to-be-played-back video data, display the target sensor data corresponding to each frame of video data of the to-be-played-back video data, and draw a ship state graph based on the target ship state data corresponding to each frame of video data of the to-be-played-back video data.

[0095] According to step 103, the data that needs to be displayed simultaneously when played is the to-be-played-back video data, the target sensor data corresponding to each frame of video data in the to-be-played-back video data, and the target ship state data; when the display terminal plays the to-be-played-back video data, the corresponding target sensor data is displayed simultaneously, and a ship state graph drawn based on the corresponding target ship state data is displayed simultaneously.

[0096] Based on the above Figure 1 As can be seen from the implementation manner of the embodiment, the present application provides a video-driven multi-component synchronous display method. Since the existing device only displays real-time data and only records a video stream, when the video is played back, the values of each sensor under the current picture cannot be fed back, and the position information of the ship cannot be displayed. Therefore, the present application obtains a target time period corresponding to to-be-played-back video data, the target time period being determined according to a creation time and an end time; traverses sensor data and ship state data stored in a database according to the creation time and the end time to obtain target sensor data and target ship state data corresponding to the target time period, wherein the sensor data and the ship state data have the same format and time stamp; determines target sensor data and target ship state data corresponding to each frame of video data of the to-be-played-back video data in the target sensor data and the target ship state data corresponding to the target time period based on the time corresponding to each frame of video data of the to-be-played-back video data using a preset correspondence rule; displays the to-be-played-back video data, displays the target sensor data corresponding to each frame of video data of the to-be-played-back video data, and draws a ship state graph based on the target ship state data corresponding to each frame of video data of the to-be-played-back video data. The present application realizes recording of sensor information, ship information, and video state information and synchronous display of video pictures, sensor data, and ship state information, thereby providing more comprehensive information parameters for offshore oil-related work.

[0097] Further, as to the Figure 1In refinement and expansion of the embodiments shown, the embodiments of the present application also provide another video-driven multi-component synchronous display method, as shown in Figure 2 The specific steps are as follows:

[0098] 201, obtain sensor data, ship state data and video data of the same time period.

[0099] Obtain a data packet containing sensor data in a preset time period; parse the data packet into plaintext data using a preset protocol to obtain the sensor data; obtain the ship state data of the preset time period; record a video corresponding to the preset time period according to a preset parameter to obtain the video data, wherein the preset parameter at least includes resolution, compression mode and frame rate.

[0100] 202, uniformly format the sensor data and ship state data at the same time according to a preset time interval, and time stamp to obtain processed sensor data and ship state data.

[0101] 203, store the processed sensor data and ship state data to a database to obtain stored sensor data and ship state data.

[0102] Examples of steps 201-203 are as follows:

[0103] The overall device is divided into an air part, a shipborne console, a monitoring terminal, the three parts are connected through wired and wireless ways, and data communication is carried out by adopting UDP protocol, in the device topology, the monitoring terminal is the server of the UDP protocol, each wireless acquisition and transmission device collects the data of each sensor through the form of serial port or network port and sends it to the interface of the monitoring terminal specified by the user role of the UDP protocol, the rest of the operation is carried out in the monitoring terminal; wherein, UDP is the abbreviation of User Datagram Protocol, and the Chinese name is User Datagram Protocol.

[0104] The specific steps are: the whole device is started, and a management and control program of the monitoring terminal is run; meanwhile, each sensor module starts to send data messages to the monitoring terminal at regular time intervals, and the monitoring terminal receives the data messages, parses the data messages into plaintext data according to the UDP protocol, and updates the sensor data values of each sensor data display area; the monitoring terminal clicks a data recording function button to trigger the whole data recording process; a video playing function starts to record a video with specified resolution (for example, 1080P), compression mode (for example, MPEG) and frame rate (for example, 30fps) parameters, and a data recording module formats the data of each sensor and the ENC chart control ship state data at the same time according to the specified time interval (the time interval is usually 1000 / recording video frame rate, unit: millisecond, if it cannot be divided, then take the integer part), timestamps are added to the data, and the data are recorded into a database; the monitoring terminal clicks a stop recording function button to end and complete the data recording. The ENC chart is an official electronic chart, which refers to a vector electronic chart made by a national hydrographic surveying and mapping agency according to the Transfer Standard for Digital Hydrographic Data (S-57) issued by the International Hydrographic Organization (IHO).

[0105] 204, obtain a target time period corresponding to the to-be-played video data, the target time period being determined according to a creation time and an end time.

[0106] This step combines the description of step 101 in the above method, and the same content is not described here.

[0107] Based on the stored to-be-played video data, the creation time and the length of the to-be-played video data are obtained; the end time is calculated according to the creation time and the length of the to-be-played video data; and the target time period is calculated according to the creation time and the end time.

[0108] For example:

[0109] The stored A to-be-played video data carries a creation time of 7:10:25, a length of 1 minute, and an end time of 7:11:25, and the target time period is 7:10:25 to 7:11:25.

[0110] 205, according to the creation time and the end time, the stored sensor data and ship state data in the database are traversed to obtain target sensor data and target ship state data corresponding to the target time period.

[0111] This step combines the description of step 102 in the above method, and the same content is not repeated here.

[0112] For example:

[0113] The stored A to be played back video data with the creation time of 7:10:25, the end time of 7:11:25, the target time period is 7:10:25 to 7:11:25; in the database with 7:10:25 to 7:11:25 as the standard to traverse the stored sensor data and ship state data, match to the start recording time and end recording time consistent with "7:10:25 to 7:11:25" sensor data and ship state data, that is, the target time period corresponding to the target sensor data and the target ship state data is obtained.

[0114] 206, based on the time corresponding to each frame of video data of the to-be-played-back video data, the target sensor data and the target ship state data corresponding to each frame of video data are determined in the target sensor data and the target ship state data corresponding to the target time period according to the preset corresponding rule.

[0115] This step combines the description of step 103 in the above method, and the same content is not repeated here.

[0116] arranging the target sensor data and the target ship state data in ascending order of time to generate a target data linked list; according to the target data linked list, matching the time corresponding to each frame of video data of the video data to be played back using a preset correspondence rule to obtain the target sensor data and the target ship state data corresponding to each frame of video data; wherein, according to the target data linked list, matching the time corresponding to each frame of video data of the video data to be played back using a preset correspondence rule to obtain the target sensor data and the target ship state data corresponding to each frame of video data, comprising: obtaining the start playing time corresponding to each frame of video data of the video data to be played back; based on the start playing time corresponding to each frame of video data, traversing the target data linked list to obtain the target sensor data and the target ship state data corresponding to each frame of video data; wherein, based on the start playing time corresponding to each frame of video data, traversing the target data linked list to obtain the target sensor data and the target ship state data corresponding to each frame of video data, comprising: obtaining the target sensor data queue and the target ship state data queue of the target data linked list, wherein the target sensor data in the target sensor data queue and the target ship state data in the target ship state data queue correspond to each other in ascending order of time, and the target sensor data and the target ship state data corresponding to each other have the same timestamp; grouping the target sensor data and the target ship state data corresponding to each other into a target data group to obtain a target data group queue, and each target data group has a unique start time; according to the start playing time corresponding to each frame of video data, traversing the target data group queue to match the target data group in which the start playing time corresponding to each frame of video data is located, and obtaining the start time of the target data group; determining whether the time difference between the start playing time corresponding to each frame of video data and the start time of the target data group is not less than a preset threshold; if not, determining that the target data group in which the start playing time corresponding to each frame of video data is located is the target data group corresponding to each frame of video data; if yes, determining the next target data group of the target data group in which the start playing time corresponding to each frame of video data is located as the target data group corresponding to each frame of video data in ascending order of time; based on the target data group, obtaining the target sensor data and the target ship state data corresponding to each frame of video data.

[0117] 207、displaying the video data to be played back, while displaying the target sensor data corresponding to each frame of video data of the video data to be played back, and drawing a ship state graph based on the target ship state data corresponding to each frame of video data of the video data to be played back.

[0118] This step combines the description of step 104 in the above method, and the same content is not repeated here.

[0119] Based on the examples from steps 201-203, steps 206 and 207 will be further explained:

[0120] The specific steps are as follows: The monitoring terminal clicks on the video data to be played back, B, to begin playback. The monitoring terminal obtains the creation time and duration of the video data to be played back, and calculates the end time. Using the creation time and end time as query conditions, it queries the database for stored sensor data and ship status data that meet the conditions. The sensor data and ship status data are then sorted in ascending order by time and loaded into memory in the form of a singly linked list. The video playback software on the monitoring terminal starts playing the first frame of the video data "video data to be played back, B", displays the sensor data in the first group of target data (which includes sensor data and ship status data) in the data linked list, and simultaneously... Based on the first set of ship status data, a ship graphic is drawn in the ENC chart control. Then, the data pointer points to the second set of target data. The video playback software on the monitoring terminal continues to play the second frame of video data "Video Data to be Played Back B". At the same time, it retrieves the data currently pointed to by the data pointer from the data chain list. If (frame time - data time) ≥ ((1000 / frame rate) / 2), then the sensor data of that frame is displayed and a ship graphic is drawn based on the ship status data of that frame. Otherwise, the data pointer moves one position to the right to point to the next frame, and the sensor data pointed to by the data pointer is displayed and the ship status is drawn based on the corresponding ship status data. After the sensor data display is completed, the data pointer moves one position to the right. The above steps are repeated until "Video Data to be Played Back B" has finished playing.

[0121] like Figure 5 As shown, when playing frame A, one set of data is displayed. When playing frame B, since t1 is less than (1000 / frame rate) / 2), the second set of data is displayed. When playing frame C, since t2 is greater than (1000 / frame rate) / 2), the third set of data is not displayed. Instead, the pointer moves to the right to display the fourth set of data. After displaying the fourth set, the data pointer moves to the right to the fifth set. When playing frame D, since the timestamp is the same as the fifth set of data, the fifth set of data is displayed, and so on.

[0122] It should be noted that when the frame rate is known, the time of each frame in the video can be obtained, and the specific method of obtaining it is not limited in this step.

[0123] It should be noted that storing video data separately from sensor data and ship status data ensures that the video data does not interfere with the sensor data and ship status data, and can provide independent functionality under specific circumstances.

[0124] Based on the above Figure 2As can be seen from the implementation mode, the application provides a video-driven multi-component synchronous display method, target sensor data and target ship state data are arranged in ascending order of time to generate a target data linked list, and target sensor data and target ship state data corresponding to each other form a target data group, a target data group queue is obtained, and each target data group has a unique start time; according to a start playing time corresponding to each frame of video data, the target data group queue is traversed, the target data group in which the start playing time corresponding to each frame of video data is located is matched, and the start time of the target data group is obtained; whether a time difference between the start playing time corresponding to each frame of video data and the start time of the target data group is not less than a preset threshold value is judged, so as to determine the target data group corresponding to each frame of video data; and the sensor data and the ship state data are uniformly formatted and marked with the timestamp, so as to obtain data pairs corresponding to the same time, so as to be matched with each frame of video data subsequently. The application overcomes the problem that the existing equipment cannot provide comprehensive data support when performing scene reproduction, experience summary and scheme deduction, and can only analyze and count through a single form of video, and the formed result lacks rigor; the application realizes recording of various sensor information, ship state information and video state information, and synchronous display of video pictures, various sensor data and ship state information by combining with an algorithm, so as to provide more comprehensive information parameters for related work.

[0125] Further, as an implementation of the method shown in the above Figure 1 , the embodiment of the application further provides a video-driven multi-component synchronous display device for implementing the method shown in the above Figure 1 . The device embodiment corresponds to the foregoing method embodiment, and for the convenience of reading, the details in the foregoing method embodiment will not be described one by one, but it should be clear that the device in the embodiment can correspondingly implement all the contents in the foregoing method embodiment. As shown in the above Figure 3 , the device comprises:

[0126] A first obtaining unit 31 is configured to obtain a target time period corresponding to the video data to be played back, and the target time period is determined according to a creation time and an end time;

[0127] A second obtaining unit 32 is configured to traverse sensor data and ship state data stored in a database according to the creation time and the end time obtained from the first obtaining unit 31, to obtain target sensor data and target ship state data corresponding to the target time period, wherein the sensor data and the ship state data have the same format and timestamp;

[0128] determining unit 33 is configured to determine, based on a time corresponding to each frame of video data of the video data to be played back, target sensor data and target ship state data corresponding to the frame of video data from the target sensor data and the target ship state data corresponding to the target time period obtained by the second obtaining unit 32 according to a preset correspondence rule;

[0129] The display unit 34 is configured to display the video data to be played back, and display the target sensor data corresponding to each frame of video data of the video data to be played back obtained from the determining unit 33, and draw a ship state graph based on the target ship state data corresponding to each frame of video data of the video data to be played back obtained from the determining unit 33.

[0130] Further, as an implementation of the method described above Figure 2 , the embodiment of the present application also provides another video-driven multi-component synchronous display device for implementing the method described above Figure 2 . The device embodiment corresponds to the foregoing method embodiment, and for the sake of reading, the details of the foregoing method embodiment will not be described one by one, but it should be clear that the device in the present embodiment can correspondingly implement all the contents in the foregoing method embodiment. As Figure 4 indicated, the device comprises:

[0131] The first obtaining unit 31 is configured to obtain a target time period corresponding to the video data to be played back, and the target time period is determined according to a creation time and an end time;

[0132] The second obtaining unit 32 is configured to traverse the stored sensor data and ship state data obtained from the storage unit 37 in the database according to the creation time and the end time obtained from the first obtaining unit 31 to obtain target sensor data and target ship state data corresponding to the target time period, wherein the sensor data and the ship state data have the same format and time stamp;

[0133] The determining unit 33 is configured to determine, based on a time corresponding to each frame of video data of the video data to be played back, target sensor data and target ship state data corresponding to the frame of video data from the target sensor data and the target ship state data corresponding to the target time period obtained by the second obtaining unit 32 according to a preset correspondence rule;

[0134] a display unit 34 configured to display the to-be-played video data, and display the target sensor data corresponding to each frame of the to-be-played video data obtained from the determination unit 33, and draw a ship state graph based on the target ship state data corresponding to each frame of the to-be-played video data obtained from the determination unit 33;

[0135] a third obtaining unit 35 configured to obtain sensor data, ship state data and video data of a same time period;

[0136] a processing unit 36 configured to uniformly format the sensor data and the ship state data of the same time obtained from the third obtaining unit 35 according to a preset time interval, and mark the time stamp to obtain processed sensor data and ship state data;

[0137] a storage unit 37 configured to store the processed sensor data and ship state data obtained from the processing unit 36 into a database to obtain stored sensor data and ship state data.

[0138] Further, the determination unit 33 comprises:

[0139] a generation module 331 configured to arrange the target sensor data and the target ship state data in ascending order of time to generate a target data linked list;

[0140] a matching module 332 configured to match the time corresponding to each frame of the to-be-played video data with a preset corresponding rule according to the target data linked list obtained from the generation module 331 to obtain the target sensor data and the target ship state data corresponding to each frame of the to-be-played video data.

[0141] Further, the matching module 332 comprises:

[0142] and further configured to obtain the start playing time corresponding to each frame of the to-be-played video data;

[0143] and further configured to traverse the target data linked list based on the start playing time corresponding to each frame of the to-be-played video data to obtain the target sensor data and the target ship state data corresponding to each frame of the to-be-played video data.

[0144] Further, the matching module 332 comprises:

[0145] The first obtaining sub-module 3321 is configured to obtain a target sensor data queue and a target ship state data queue of the target data link table, wherein the target sensor data in the target sensor data queue and the target ship state data in the target ship state data queue correspond to each other in ascending order of time, and the target sensor data and the target ship state data corresponding to each other have the same time stamp;

[0146] The composing sub-module 3322 is configured to compose the target sensor data and the target ship state data corresponding to each other obtained from the first obtaining sub-module 3321 into a target data group to obtain a target data group queue, and each target data group has a unique start time;

[0147] The matching sub-module 3323 is configured to traverse the target data group queue obtained from the composing sub-module 3322 according to the start playing time corresponding to each frame of video data, match the target data group in which the start playing time corresponding to each frame of video data is located, and obtain the start time of the target data group;

[0148] The judging sub-module 3324 is configured to judge whether the time difference between the start playing time corresponding to each frame of video data and the start time of the target data group obtained from the matching sub-module 3323 is not less than a preset threshold value;

[0149] The determining sub-module 3325 is configured to, if the time difference between the start playing time corresponding to each frame of video data and the start time of the target data group obtained from the judging sub-module 3324 is less than the preset threshold value, determine that the target data group in which the start playing time corresponding to each frame of video data is located is the target data group corresponding to each frame of video data;

[0150] The determining sub-module 3325 is further configured to, if the time difference between the start playing time corresponding to each frame of video data and the start time of the target data group obtained from the judging sub-module 3324 is not less than the preset threshold value, determine, in ascending order of time, that the next target data group of the target data group in which the start playing time corresponding to each frame of video data is located is the target data group corresponding to each frame of video data;

[0151] The second obtaining sub-module 3326 is configured to obtain the target sensor data and the target ship state data corresponding to each frame of video data based on the target data group obtained from the determining sub-module 3325.

[0152] Further, the third obtaining unit 35 comprises:

[0153] The first obtaining module 351 is configured to obtain data messages containing sensor data in a preset time period;

[0154] a parsing module 352, configured to parse the data packet obtained from the first obtaining module 351 into plaintext data by using a preset protocol, to obtain the sensor data;

[0155] a second obtaining module 353, configured to obtain the ship state data of the preset time period obtained from the first obtaining module 351;

[0156] a third obtaining module 354, configured to record a video corresponding to the preset time period obtained from the first obtaining module 351 according to preset parameters, to obtain the video data, wherein the preset parameters at least include resolution, compression mode and frame rate.

[0157] Further, the first obtaining unit 31 comprises:

[0158] an obtaining module 311, configured to obtain the creation time and duration of the to-be-played video data based on the stored to-be-played video data;

[0159] a first calculating module 312, configured to calculate the corresponding end time according to the creation time and duration of the to-be-played video data obtained from the obtaining module 311;

[0160] a second calculating module 313, configured to calculate the corresponding target time period according to the creation time and end time obtained from the first calculating module 312.

[0161] Further, the embodiment of the present application also provides a processor, which is used for running a program, wherein the program performs the video-driven multi-component synchronous display method described in the above Figures 1-2 when running.

[0162] Further, the embodiment of the present application also provides a storage medium, which is used for storing a computer program, wherein the computer program controls a device where the storage medium is located to perform the video-driven multi-component synchronous display method described in the above Figures 1-2 when running.

[0163] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0164] It can be understood that the related features in the above method and device can be mutually referred. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, and do not represent the advantages and disadvantages of the embodiments.

[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, the device and the unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0166] The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description above. In addition, the present application is not intended to be limited to any particular programming language. It will be appreciated that there are many programming languages that can be used to implement the teachings of the present application as described herein, and any such programming language can be used in connection with the various aspects of the present application.

[0167] In addition, the storage can include non-transitory storage such as a storage medium in a computer readable medium, a Random Access Memory (RAM), and / or a non-volatile memory such as a Read Only Memory (ROM) or a flash memory, and the storage includes at least one memory chip.

[0168] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0169] The present application is described with reference to the drawings in which are shown flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present application. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing element or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 The flowchart and / or block diagram in the flowcharts and / or block diagrams can represent one or more of any flow processes, functions and / or operations described and / or claimed in the specification and drawings. Figure 1 The flowchart and / or block diagram in the flowcharts and / or block diagrams can represent one or more of any flow processes, functions and / or operations described and / or claimed in the specification and drawings.

[0170] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowcharts and / or block diagrams block or blocks. Figure 1 The flowchart and / or block diagram in the flowcharts and / or block diagrams can represent one or more of any flow processes, functions and / or operations described and / or claimed in the specification and drawings.Figure 1 the function(s) specified in the block or blocks.

[0171] These computer program instructions can also be loaded into computer or other programmable data processing devices to cause a series of operational steps to be performed on the computer or other programmable devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable devices provide steps for implementing the flowchart block(s) or flowchart flow(s) and / or portions thereof. Figure 1 the flowchart block(s) or flowchart flow(s) and / or portions thereof. Figure 1 the function(s) specified in the block or blocks.

[0172] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0173] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) about which the computer stores information such as computer program code and data structures; and non-volatile memory, such as read only memory (ROM), EPROM, EEPROM, or flash memory, about which the computer stores information, such as computer program code and data structures. Memory is an example of computer readable media.

[0174] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile discs (DVD) or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to computing devices. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0175] It should also be noted that the terms "comprising," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements in the list, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without limitation, an element preceded by "comprises a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0176] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code thereon for use by or in connection with an instruction execution system. For the purposes of this description, a computer-usable or computer readable storage medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0177] The foregoing is merely illustrative of the embodiments of this application, and is not intended to limit the application. Numerous variations and modifications can be possible to the embodiments without departing from the spirit and scope of the application. Any equivalent modifications or variations, made within the spirit and scope of the application, should be considered within the scope of the application.

Claims

1. A method for video-driven multi-component synchronized display, characterized in that, The method is applied to the offshore oil cleaning scene, and comprises: obtaining a target time period corresponding to to-be-replayed video data, the target time period being determined according to a creation time and an end time; traversing sensor data and ship state data stored in a database according to the creation time and the end time to obtain target sensor data and target ship state data corresponding to the target time period, wherein the sensor data and the ship state data have the same format and time stamp; determining target sensor data and target ship state data corresponding to each frame of video data of the to-be-replayed video data in the target sensor data and the target ship state data corresponding to the target time period based on a time corresponding to each frame of video data of the to-be-replayed video data using a preset correspondence rule, the determining target sensor data and target ship state data corresponding to each frame of video data of the to-be-replayed video data in the target sensor data and the target ship state data corresponding to the target time period based on a time corresponding to each frame of video data of the to-be-replayed video data using a preset correspondence rule comprising: arranging the target sensor data and the target ship state data in ascending order of time to generate a target data linked list; matching the time corresponding to each frame of video data of the to-be-replayed video data using the preset correspondence rule according to the target data linked list to obtain the target sensor data and the target ship state data corresponding to each frame of video data; displaying the to-be-replayed video data, simultaneously displaying the target sensor data corresponding to each frame of video data of the to-be-replayed video data, and drawing a ship state graph based on the target ship state data corresponding to each frame of video data of the to-be-replayed video data.

2. The method of claim 1, wherein, The matching the time corresponding to each frame of video data of the to-be-replayed video data using the preset correspondence rule according to the target data linked list to obtain the target sensor data and the target ship state data corresponding to each frame of video data comprises: obtaining a start playing time corresponding to each frame of video data of the to-be-replayed video data; traversing the target data linked list based on the start playing time corresponding to each frame of video data to obtain the target sensor data and the target ship state data corresponding to each frame of video data.

3. The method of claim 2, wherein, The traversing the target data linked list based on the start playing time corresponding to each frame of video data to obtain the target sensor data and the target ship state data corresponding to each frame of video data comprises: obtaining a target sensor data queue and a target ship state data queue of the target data linked list, wherein the target sensor data in the target sensor data queue and the target ship state data in the target ship state data queue correspond to each other in ascending order of time, and the target sensor data and the target ship state data corresponding to each other have the same time stamp; grouping the target sensor data and the target ship state data corresponding to each other into target data groups to obtain a target data group queue, and each target data group has a unique start time; traversing the target data group queue according to the start playing time corresponding to each frame of video data, matching the target data group in which the start playing time corresponding to each frame of video data is located, and obtaining the start time of the target data group; determining whether the time difference between the start playing time corresponding to each frame of video data and the start time of the target data group is not less than a preset threshold value; if not, determining that the target data group in which the start playing time corresponding to each frame of video data is located is the target data group corresponding to each frame of video data; if yes, determining the next target data group of the target data group in which the start playing time corresponding to each frame of video data is located as the target data group corresponding to each frame of video data in ascending order of time; based on the target data group, obtaining target sensor data and target ship state data corresponding to each frame of video data.

4. The method of claim 1, wherein, Before the target time period corresponding to the video data to be played back is obtained, the target time period is determined according to the creation time and the end time, the method comprises: obtaining sensor data, ship state data and video data of the same time period; unifying the format of the sensor data and the ship state data at the same time according to a preset time interval, and marking the timestamp to obtain processed sensor data and ship state data; storing the processed sensor data and ship state data to a database to obtain stored sensor data and ship state data.

5. The method of claim 4, wherein, The obtained sensor data, ship state data and video data of the same time period comprise: obtaining a data packet containing sensor data of a preset time period; parsing the data packet into plaintext data by using a preset protocol to obtain the sensor data; obtaining the ship state data of the preset time period; recording a video corresponding to the preset time period according to a preset parameter to obtain the video data, wherein the preset parameter at least comprises resolution, compression mode and frame rate.

6. The method according to any one of claims 1-5, characterized in that, The target time period corresponding to the video data to be played back is obtained, and the target time period is determined according to the creation time and the end time, which comprises: obtaining the creation time and the length of the video data to be played back based on the stored video data to be played back; calculating the corresponding end time according to the creation time and the length of the video data to be played back; calculating the corresponding target time period according to the creation time and the end time.

7. A monitoring device for the state of an object, characterized in that The device is applied to a marine oil cleaning scene, and comprises: a first obtaining unit configured to obtain a target time period corresponding to video data to be played back, the target time period being determined according to a creation time and an end time; a second obtaining unit configured to traverse stored sensor data and ship state data in a database according to the creation time and the end time to obtain target sensor data and target ship state data corresponding to the target time period, wherein the sensor data and the ship state data have the same format and timestamp; A determination unit is configured to determine target sensor data and target ship state data corresponding to each frame of video data of the to-be-played-back video data based on a preset correspondence rule and the target sensor data and the target ship state data corresponding to the target time period. A generation module is configured to generate a target data linked list by arranging the target sensor data and the target ship state data in ascending order of time. A matching module is configured to match the time corresponding to each frame of video data of the to-be-played-back video data with the preset correspondence rule based on the target data linked list obtained from the generation module, to obtain the target sensor data and the target ship state data corresponding to each frame of video data. A display unit is configured to display the to-be-played-back video data, display the target sensor data corresponding to each frame of video data of the to-be-played-back video data, and draw a ship state graph based on the target ship state data corresponding to each frame of video data of the to-be-played-back video data.

8. A storage medium, the storage medium comprising a stored program, characterized in that The program controls a device where the storage medium is located to execute the video-driven multi-component synchronous display method in any one of claims 1 to 6 when the program is running.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the video-driven multi-component synchronous display method in any one of claims 1 to 6 when executing the program.

Images