An interactive method, device and computer equipment based on multi-feature recognition

Abstract

The present invention discloses an interactive method, device and computer equipment based on multi-feature recognition, wherein the method includes: obtaining target video stream data of a target device; calling a three-dimensional model of the target device according to the target video stream data; sending the data of the three-dimensional model to a remote device; receiving the change increment information of the three-dimensional model fed back by the remote device; and displaying the change of the three-dimensional model according to the change increment information. By implementing the present invention, the problem that the remote repair and maintenance operations existing in the related technology cannot be accurate and effective is solved. By combining the three-dimensional model generated according to the target video stream data and the change increment information fed back by the received remote device, the on-site equipment can obtain accurate guidance information, and realize the remote virtual-real fusion interaction of the augmented reality mode and the three-dimensional model of the power equipment.

Description

Technical Field

[0001] The present invention relates to the fields of streaming media and information communication, and in particular to an interaction method, device and computer equipment based on multi-feature recognition. Background Art

[0002] With the development of the Internet, information communication and 5G communication technology, audio and video streaming call technology has mainly gone through four development stages: local analog signal audio and video system, PC-based multimedia remote assistance system, Web server-based remote audio and video collaboration system, and mobile-based audio and video collaboration system. At present, the development of audio and video streaming call technology is in the stage of mobile-based audio and video collaboration system (audio and video call).

[0003] With the development of science and technology, higher requirements have been put forward for the collaboration mode. The collaboration mode of remote video calls in related technologies still has great limitations. Specifically, as a public service enterprise related to energy security and national economy and people's livelihood, power companies have operation sites such as power grid transmission and distribution inspection, maintenance, emergency repair, etc. There are many types of power equipment and the operation is complex. New problems are constantly emerging and difficult to identify and handle. When handling, cross-team and cross-work area collaborative work and remote support from technical experts or equipment manufacturers are required. In related technologies, the way to provide remote support is mostly through ordinary video calls. Since the perspective of the video collected by the remote video call equipment is relatively limited, it is likely that real-time guidance and operation of the site from the same perspective cannot be achieved, resulting in remote maintenance operations not being very accurate and effective. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide an interactive method, apparatus and computer device based on multi-feature recognition to solve the problem that remote repair and maintenance operations in related technologies are not very accurate and effective.

[0005] According to the first aspect, an embodiment of the present invention provides an interaction method based on multi-feature recognition, including: obtaining target video stream data of a target device; calling a three-dimensional model of the target device according to the target video stream data; sending data of the three-dimensional model to a remote device; and receiving incremental change information of the three-dimensional model fed back by the remote device.

[0006] In combination with the first aspect, in a first implementation of the first aspect, the method further includes: displaying changes of the three-dimensional model according to the change increment information; and controlling the target device according to the changes of the three-dimensional model.

[0007] In combination with the first aspect, in the second implementation of the first aspect, the acquisition of target video stream data of the target device includes: collecting and sending initial video stream data of the target device in the target area; receiving an initial key frame sent by the remote device; determining a target key frame based on the initial key frame; and acquiring the target video stream data of the target device based on the target key frame.

[0008] In combination with the second implementation of the first aspect, in the third implementation of the first aspect, determining the target key frame based on the initial key frame includes: extracting color feature information, texture feature information and motion feature information in the initial video key frame; fusing the color feature information, texture feature information and motion feature information, and respectively calculating the similarity of each initial video key frame; determining candidate video key frames based on the similarity of each initial video key frame; and determining the target key frame based on a preset adaptive algorithm.

[0009] In combination with the third implementation manner of the first aspect, in the fourth implementation manner of the first aspect, obtaining the target video stream data of the target device based on the target key frame includes: obtaining first video stream data; identifying a first feature point in the first video stream data according to a preset optical flow method, and identifying a second feature point in the target key frame; when the similarity between the first feature point and the second feature point is greater than a preset similarity threshold, determining that the first video stream data matches the target key frame; when the first video stream data matches the target key frame, determining the first video stream as the target video stream data of the target device.

[0010] In combination with the first aspect, in the fifth implementation of the first aspect, the method also includes: determining the first center position of the target key frame based on the first feature point and the preset relative distance; determining the second center position of the first video stream data based on the second feature point and the preset relative distance; and tracking and acquiring the target video stream data of the target device based on the first center position and the second center position.

[0011] According to the second aspect, an embodiment of the present invention provides an interactive method based on multi-feature recognition, including: receiving data of a three-dimensional model sent by a field device; generating a three-dimensional model based on the data of the three-dimensional model; determining change increment information based on the three-dimensional model and a preset database; and feeding back the change increment information to the field device.

[0012] In combination with the second aspect, in the first implementation of the second aspect, before receiving the data of the three-dimensional model sent by the on-site device, it also includes: receiving the initial video stream data of the target area sent by the on-site device; determining the problem area based on the initial video stream data, and generating an initial key frame based on the problem area; and sending the initial key frame to the on-site device.

[0013] According to the third aspect, an embodiment of the present invention provides an interactive device based on multi-feature recognition, including: a target video stream data acquisition module, used to acquire target video stream data of a target device; a calling module, used to call a three-dimensional model of the target device according to the target video stream data; a data sending module, used to send the data of the three-dimensional model to a remote device; a change increment information receiving module, used to receive the change increment information of the three-dimensional model fed back by the remote device.

[0014] According to the fourth aspect, an embodiment of the present invention provides an interactive device based on multi-feature recognition, including: a data receiving module for receiving data of a three-dimensional model sent by an on-site device; a three-dimensional model generating module for generating a three-dimensional model based on the data of the three-dimensional model; a determination module for determining change increment information based on the three-dimensional model and a preset database; and a change increment information sending module for feeding back the change increment information to the on-site device.

[0015] According to the fifth aspect, an embodiment of the present invention provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the interaction method based on multi-feature recognition described in the first aspect or any one embodiment of the first aspect, or the steps of the interaction method based on multi-feature recognition described in the second aspect or any one embodiment of the second aspect.

[0016] According to the sixth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the interaction method based on multi-feature recognition described in the first aspect or any one of the embodiments of the first aspect are implemented, or the steps of the interaction method based on multi-feature recognition described in the second aspect or any one of the embodiments of the second aspect are implemented.

[0017] The technical solution of the present invention has the following advantages:

[0018] 1. An interactive method, device and computer device based on multi-feature recognition provided by an embodiment of the present invention, wherein the method comprises: obtaining target video stream data of a target device; calling a three-dimensional model of the target device according to the target video stream data; sending the data of the three-dimensional model to a remote device; receiving the change increment information of the three-dimensional model fed back by the remote device; and controlling the target device according to the change increment information. By implementing the present invention, in combination with the three-dimensional model generated according to the target video stream data and the change increment information fed back by the remote device, the on-site equipment can obtain accurate guidance information, and realize the remote virtual-real fusion interaction of the augmented reality mode and the three-dimensional model of the power equipment.

[0019] 2. An interactive method, device and computer equipment based on multi-feature recognition provided by an embodiment of the present invention, wherein the method comprises: receiving data of a three-dimensional model sent by a field device; generating a three-dimensional model according to the data of the three-dimensional model; determining change increment information according to the three-dimensional model and a preset database; and feeding back the change increment information to the field device. By implementing the present invention and combining the generated three-dimensional model, the remote device can mark the target device in a first-person perspective, thereby accurately guiding the field workers to perform their operations, which is efficient and accurate, and realizes the remote virtual-real fusion interaction of the augmented reality method and the three-dimensional model of the power equipment.

[0020] 3. An interactive method based on multi-feature recognition provided by an embodiment of the present invention combines collaborative labeling and identification matching between a remote device and an on-site device, determines the target position by the relative distance of feature points, and can continuously detect the features of the target device, thereby continuously obtaining the real-time position of the target device and achieving accurate tracking of the target device. That is to say, the multi-feature point information of the target device is detected in real time and matched with the temporarily stored video key frames to achieve monitoring, identification, matching and tracking of the target device. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a structural diagram of communication between a field device and a remote device in an interactive method based on multi-feature recognition in an embodiment of the present invention;

[0023] Figure 2It is a flowchart of a specific example of a field device end in an interactive method based on multi-feature recognition in an embodiment of the present invention;

[0024] Figure 3 It is a flowchart of a specific example of obtaining target video stream data in the interactive method based on multi-feature recognition in an embodiment of the present invention;

[0025] Figure 4 It is a flowchart of a specific example of a remote device end in an interactive method based on multi-feature recognition in an embodiment of the present invention;

[0026] Figure 5 is a flowchart of another specific example of a remote device end in an interactive method based on multi-feature recognition in an embodiment of the present invention;

[0027] Figure 6 It is a structural block diagram of a specific embodiment of the interactive method based on multi-feature recognition in an embodiment of the present invention;

[0028] Figure 7 It is a flowchart of a specific example of a field device end of an interactive device based on multi-feature recognition in an embodiment of the present invention;

[0029] Figure 8 It is a principle block diagram of a specific example of a remote device end in an interactive device based on multi-feature recognition in an embodiment of the present invention;

[0030] Fig. 9 FIG. 4 is a specific example diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The technical features involved in the different implementation modes of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0032] Collaboration is the trend of modern social development. Traditional face-to-face collaboration methods (such as video conferencing) have huge limitations in both time and space, and are far from meeting people's requirements for collaboration. Specifically, remote collaboration refers to the process of helping geographically dispersed organizations and individuals complete collaboration with the support of computer and communication technology. In order to support effective collaboration, the platform must be able to support live streaming media such as real-time video and audio, and must also support other multimedia information, such as graphic annotations, static images, text, etc., as well as comprehensive processing of these multimedia information. In actual application scenarios, for example, at the operation sites of power grid transmission and distribution inspection, maintenance, emergency repair, etc., there are many types of power equipment and complex operations, new problems continue to emerge and are difficult to identify and handle, requiring cross-team and cross-work area collaborative operations, as well as remote technical support from technical experts or equipment manufacturers. However, the current communication method is inefficient and has a high probability of error, which needs to be solved urgently.

[0033] In order to solve the problems of low communication efficiency and high error probability existing in related technologies, an embodiment of the present invention provides an interaction method, device and computer equipment based on multi-feature recognition, the purpose of which is to efficiently and accurately guide the operation of the on-site equipment through real-time audio and video interaction between the on-site equipment and the remote equipment.

[0034] like Figure 1 As shown, the on-site device and the remote expert device communicate through a wireless channel, wherein the on-site device may be provided with a wearable terminal device, a camera device or a control ball and other image acquisition devices, and a wireless communication device; the remote device may be provided with a wireless communication module for communicating with other devices, and the remote device may be a remote expert device. Specifically, the on-site device may send the collected on-site video stream data to the remote expert device, and the remote expert device may receive the video stream data through the wireless communication module and send back corresponding feedback information.

[0035] The embodiment of the present invention provides an interactive method based on multi-feature recognition, which is specifically applied to a field device end, such as Figure 2 As shown, including:

[0036] Step S11: Obtain target video stream data of the target device; in this embodiment, the target device can be any device provided in the actual application scenario. For example, when applied to the power grid transmission scenario, the target device can be an electronic device, such as an oil temperature gauge, an electronic switch, etc. Video stream data can be a data form that stores and records a series of continuous image information. The continuous images record specific events in one or more continuous time periods. When the continuous images are played sequentially at a faster frame rate, a continuous picture will be displayed, which is the video stream data. The target video stream data can be the video stream data of the corresponding problem area obtained again based on the video stream data of the problem area marked by the remote device.

[0037] Specifically, the on-site device obtains the video stream data of the problem area, which is the target video stream data. For example, when the remote device (e.g., the technical support expert terminal) marks the device with the problem as an oil temperature gauge, when the camera of the on-site device moves to point to the oil temperature gauge, the video stream data containing the oil temperature gauge can be obtained, which is the target video stream data.

[0038] Specifically, the on-site equipment can obtain video stream data through wearable terminal devices, camera equipment or surveillance balls.

[0039] Step S12: calling the three-dimensional model of the target device according to the target video stream data; in this embodiment, the three-dimensional model may refer to a three-dimensional stereoscopic model, which is used to characterize the structural feature information of the target device, etc. Specifically, at the on-site device end, according to the acquired target video stream data containing the target device, the identification information of the target device in the target video stream data is determined, the model of the target device is determined according to the identification information, and the corresponding three-dimensional model is called in the pre-set three-dimensional model database according to the model of the target device. For example, when the target device is determined to be an oil temperature gauge according to the target video stream data, the device model of the oil temperature gauge is first determined, such as xxx-1, and then the three-dimensional model of the oil temperature gauge with the device model of xxx-1 is called in the pre-set three-dimensional model data and displayed at the on-site device end.

[0040] Step S13: Send the data of the three-dimensional model to the remote device; in this embodiment, the on-site device and the remote expert communicate and transmit data via a wireless channel; the on-site device sends the generated three-dimensional model data to the remote device.

[0041] Step S14: receiving the incremental change information of the three-dimensional model fed back by the remote device; in this embodiment, through step S13, the three-dimensional model of the target device is displayed on the remote device based on the data of the three-dimensional model. The incremental change information of the three-dimensional model can be an information record of the operation performed on the three-dimensional model of the target device in order to solve the problem existing in the target device when the remote expert end observes the target device from the first perspective. For example, when the remote expert end confirms that there is a problem with the oil temperature gauge, it will perform corresponding operations on the oil temperature gauge on the three-dimensional model of the remote expert end, for example, move the oil temperature gauge to the left by 0.6 cm. At this time, the incremental change information is "move the oil temperature gauge to the left by 0.6 cm". The above incremental change information "move the oil temperature gauge to the left by 0.6 cm" is transmitted from the remote expert end to the on-site device through a wireless channel.

[0042] An interactive method based on multi-feature recognition provided by an embodiment of the present invention includes: obtaining target video stream data of a target device; calling a three-dimensional model of the target device according to the target video stream data; sending the data of the three-dimensional model to a remote device; receiving the change increment information of the three-dimensional model fed back by the remote device; and displaying the change of the three-dimensional model according to the change increment information. By implementing the present invention, combined with the three-dimensional model generated according to the target video stream data and the change increment information fed back by the remote device, the on-site equipment can obtain accurate guidance information, and realize the remote virtual-real fusion interaction of the augmented reality mode and the three-dimensional model of the power equipment.

[0043] As an optional embodiment of the present invention, the method further includes: displaying the change of the three-dimensional model according to the change increment information; and controlling the target device according to the change of the three-dimensional model. In this embodiment, the field device can display the change increment information directly on the three-dimensional model. For example, according to the received change increment information, the three-dimensional model is adjusted accordingly. For example, when the received change increment information is "move the oil temperature gauge to the left by 0.6 cm", the oil temperature gauge in the three-dimensional model at the field device end directly moves to the left by 0.6 cm. Furthermore, the operation and maintenance personnel can control the target device according to the change of the three-dimensional model. For example, the operation and maintenance personnel can control the oil temperature gauge in the actual device according to the change of the three-dimensional model, so that the oil temperature gauge moves to the left by 0.6 cm.

[0044] As an optional implementation of the present invention, the above step S11, obtaining the target video stream data of the target device, includes:

[0045] Step S21: Collect and send the initial video stream data of the target device in the target area; in this embodiment, the target area can be any area in the actual application scenario, and the initial video stream data can be the video stream initially collected by the on-site device end. At this time, the video stream data of the on-site device end can be collected through a wearable terminal device, a camera device or a control ball, and the collected initial video stream data can be transmitted in real time to the remote device end, that is, the remote expert end.

[0046] Step S22: receiving the initial key frame sent by the remote device; in the present embodiment, the initial key frame may be a drawing annotation of the problematic area in the initial video stream data by the remote expert end from the first perspective, such as text annotation and image annotation, etc., and the video stream segment with the drawing annotation is the initial key frame; specifically, after sending the initial video stream data to the remote expert end, the on-site device may receive the initial key frame fed back by the remote expert end.

[0047] Step S23: Determine the target key frame according to the initial key frame; in the present embodiment, the target key frame may be a frame containing all feature information in the initial key frame extracted after the on-site equipment optimizes and aggregates the initial key frame, that is, the target key frame; specifically, extract the key information in the initial key frame, eliminate the redundant information in the initial key frame, extract the frame with the key information as the target key frame through image saliency detection method, candidate key frame extraction method, adaptive hierarchical clustering and other methods, and store the target key frame in a structured manner.

[0048] Step S24: According to the target key frame, the target video stream data of the target device is obtained. In this embodiment, according to the target key frame, the target video stream data of the target device is determined, and the target key frame can be matched with the re-collected video stream data; when the re-collected video stream data matches the target key frame, it can be determined that the re-collected video stream data is the target video stream data, and at this time, the text annotation, image annotation, and three-dimensional model annotation of the remote expert end on the target key frame can be displayed on the re-collected video stream data.

[0049] An interactive method based on multi-feature recognition provided by an embodiment of the present invention is combined with video stream key frame technology to extract key information in the video, eliminate redundant information in the video, determine the target key frame through image saliency detection, candidate key frame extraction, adaptive hierarchical clustering and other methods, and store the above target key frame in a structured manner. The target key frame and target video stream data can be determined efficiently and accurately, thereby minimizing resource consumption and maximizing key information storage.

[0050] As an optional implementation of the present invention, the above step S23, the execution process of determining the target key frame according to the initial key frame, includes:

[0051] First, extract the color feature information, texture feature information and motion feature information from the initial video key frame; in this embodiment, the color feature information is the most significant feature in the image, which is based on the characteristics of the pixel points. Different electrical devices display different colors. The process of extracting color feature information may include extracting the color feature information from the initial video key frame and describing the above color feature information with a histogram. Specifically, a color histogram may be generated based on the different color feature information of each electrical device. Texture features are global features of an image, which describe the surface properties of the scene corresponding to the image or image region; the process of extracting texture feature information may include obtaining texture feature information by performing statistical calculations based on multiple regions containing multiple pixels; specifically, the initial video key frame is divided into multiple images through a Markov random field model to obtain texture feature information such as the number of different regions, pixel positions, and pixel value sets, and determine the texture feature information of the initial video key frame. The process of extracting motion feature information may include first extracting the salient image of the initial video key frame, specifically, by using a saliency detection algorithm (SDSP) and core rules based on CIEL*a*b* color features, contrast principle, and saliency calculation, determining the salient target in the initial video key frame, and preserving most of the information of the original image. Secondly, by using a pyramid-based Lucas-Kanade optical flow method, motion estimation is performed on the saliency image to generate motion feature information of the saliency image.

[0052] The embodiment of the present invention provides an interactive method based on multi-feature recognition, combined with the SDSP algorithm, that is, combined with three prior knowledge: the behavior of human vision in detecting prominent objects in the scene, which can be simulated by log-Gabor filter; human vision tends to focus on the center of the image, which is modeled by Gaussian mapping; warm colors attract more visual attention than cold colors. Mathematical modeling can be used to eliminate the influence of color, complex texture and changing background, and quickly and accurately obtain a salient image.

[0053] Secondly, the color feature information, texture feature information and motion feature information are fused, and the similarity of each initial video key frame is calculated respectively; in this embodiment, the similarity between each initial video key frame can be the degree of the sameness of the content contained in each initial video key frame. Specifically, the color feature information, texture feature information and motion feature information are normalized to generate a fused feature vector. According to the fused feature vector, the Euclidean distance between two adjacent initial video key frames is calculated, and then the similarity between two adjacent initial video key frames is calculated. The smaller the Euclidean distance, the higher the similarity between two adjacent frames.

[0054] Secondly, according to the similarity of each initial video key frame, the candidate video key frame is determined; according to the preset adaptive algorithm, the target key frame is determined. In this embodiment, the clustering threshold is determined by the adaptive hierarchical clustering algorithm, and the mutual information (MI) between the salient images of each initial video key frame is determined, and the mutual information is used to characterize the correlation between two variables. Specifically, the process of determining the target key frame by the adaptive hierarchical clustering algorithm can be: according to the initial video key frame, the candidate video key frame is determined, that is, the candidate key frame sequence; the mutual information between the salient images of each adjacent candidate key frame sequence is calculated, and the mutual information sequence is calculated and determined; according to the normalized overlapping area and histogram of each adjacent image, the joint probability is calculated, and the clustering threshold is determined according to the joint probability; the mutual information sequence is arranged in descending order according to the mutual information value, and then according to the original time order of the candidate key frame, the first frame is taken as the first cluster, and if the mutual information value between the subsequent two frames is less than or equal to the threshold, a new cluster is generated. On the contrary, subsequent frames are divided into the current cluster to determine the target key frame, and the target key frame is an ordered cluster and the frames in each cluster are also arranged in order according to the relevance of the original video content.

[0055] An interactive method based on multi-feature recognition provided by an embodiment of the present invention, combined with the rotational invariance of texture feature information, can have a strong resistance to noise, thereby realizing the distinction of object information contained in the image at a microscopic level. In combination with image saliency detection, extraction of candidate key frames, and key frames determined by clustering adaptive algorithms, the SDSP algorithm can be applied to the original video sequence, and the saliency information that attracts the attention of the human eye in the video can be extracted by combining three saliency detection methods based on prior knowledge; the data information contained in the video frame can be quantitatively described, and then candidate key frames with less redundancy can be obtained; the threshold is determined by clustering adaptively, which better solves the problem of unstable clustering results caused by inaccurate initial boundary point selection. The final clusters obtained after adaptive hierarchical clustering are arranged in the chronological order of the original video content, and the extracted key frames maintain the timing of the original input video.

[0056] As an optional implementation of the present invention, step S24, the execution process of acquiring the target video stream data of the target device according to the target key frame, includes:

[0057] First, the first video stream data is obtained; in this embodiment, it can be the video stream data collected when the wearable terminal device, the camera device or the control ball moves to the target area again.

[0058] Secondly, the first feature point in the first video stream data is identified according to the preset optical flow method, and the second feature point in the target key frame is identified; in this embodiment, the first feature point in the first video stream data and the second feature point in the target key frame are determined by the forward optical flow method or the backward optical flow method. Specifically, the first feature point can be a plurality of feature points including a plurality of features in the first video stream data, for example, it can be a target feature point in a pixel; then the target key frame is queried, and a plurality of target feature points in the target key frame are extracted.

[0059] Secondly, when the similarity between the first feature point and the second feature point is greater than a preset similarity threshold, it is determined that the first video stream data matches the target key frame; in this embodiment, the similarity between the first feature point in the first video stream data and the second feature point in the target key frame in terms of position, quantity, etc. is calculated and compared with the preset similarity threshold. When the calculated similarity is greater than the preset similarity threshold, it is determined that the first video stream data matches the target key frame successfully.

[0060] Secondly, when the first video stream data matches the target key frame, the first video stream is determined as the target video stream data of the target device. In this embodiment, when the first video stream data successfully matches the target key frame data, it means that the on-site device has recaptured the video stream data of the target device marked as problematic by the remote expert through the wearable terminal, which is the target video stream data.

[0061] An embodiment of the present invention provides an interactive method based on multi-feature recognition. When applied to power sites and emergency repair operation scenarios, the video stream of the on-site equipment can be collected through a wearable terminal, a camera device or a control ball, and then multiple frames in the video stream are read. The remote expert draws and annotates the collaborative targets of the video stream collected by the on-site workers from a first-person perspective, identifies the target feature points through the forward / backward optical flow method, calculates the characteristics and feature description of the current frame, queries the key frame temporary storage set, matches the feature points in the current frame with the feature points of the key frame temporary storage set, and if the match is successful, the remote collaborative annotation target recognition match is successful, preparing for augmented reality information overlay interaction, otherwise the key frame increment is updated and stored in the key frame temporary storage set. That is to say, when the first video stream data successfully matches the target key frame, a three-dimensional model can be generated according to the power equipment based on the augmented reality service platform and augmented reality method, and text and image annotations can be superimposed on the three-dimensional model. Through real-time collaboration, the position relationship, angle, operation behavior and model feedback results between personnel, power equipment and models, the incremental change information is transmitted and encoded and decoded at the distributed terminals. At the same time, a verification mechanism is used to ensure the synchronous change of the model and information, so as to realize the tracking interaction between the on-site equipment and the remote expert terminal based on multi-feature recognition.

[0062] As an optional embodiment of the present invention, the method further comprises:

[0063] According to the first feature point and the preset relative distance, the first center position of the target key frame is determined; according to the second feature point and the preset relative distance, the second center position of the first video stream data is determined; according to the first center position and the second center position, the target video stream data of the target device is tracked and acquired. In this embodiment, the preset relative distance is the distance of the target feature point based on the center position. Since the relative distance between the feature point and the center position remains unchanged during scaling and rotation of the same image, the first center position of the target key frame is determined according to the first feature point and the preset relative distance, and the center position of the first video stream data is determined according to the second feature point; according to the detected center position, the target video stream data of the target device is continuously acquired to achieve continuous tracking of the target device.

[0064] The embodiment of the present invention provides an interactive method based on multi-feature recognition, which combines cluster voting on the center to determine the center position and the relative distance of each feature point to determine the position of the target device. Since the distance of each feature point relative to the center position is determined under the scaling and rotation ratio, the real-time tracking of the object position can be achieved by continuously detecting the object features. The monitoring, identification, matching and tracking of the object is achieved by real-time detection of the multi-feature point information of the object and matching it with the temporary storage set of the structured key frame of the video stream.

[0065] The embodiment of the present invention provides an interactive method based on multi-feature recognition, which is specifically applied to a remote device, such as Figure 4 As shown, including:

[0066] Step S31: receiving the data of the three-dimensional model sent by the on-site device; in this embodiment, the remote expert terminal receives the data of the three-dimensional model sent by the on-site device terminal.

[0067] Step S32: Generate a three-dimensional model according to the data of the three-dimensional model; in this embodiment, the remote expert terminal constructs the three-dimensional model according to the received three-dimensional model data.

[0068] Step S33: Determine the incremental change information based on the three-dimensional model and the preset database; in this embodiment, the remote expert terminal adjusts the problematic area in the three-dimensional model according to the preset database. For example, when the remote expert terminal determines that there is a problem with the oil temperature gauge in the three-dimensional model, it adjusts the oil temperature gauge according to the preset database, for example, moving the oil temperature gauge 10 cm to the left or 0.6 cm to the left. The above adjustment information is the incremental change information.

[0069] Step S34: Feedback the change increment information to the field device. In this embodiment, when the change increment information is 0.6 cm to the left, the adjustment information "adjust the oil temperature gauge to 0.6 cm" is transmitted to the field device as the change increment information.

[0070] An interactive method based on multi-feature recognition provided by an embodiment of the present invention includes: receiving data of a three-dimensional model sent by a field device; generating a three-dimensional model according to the data of the three-dimensional model; determining change increment information according to the three-dimensional model and a preset database; and feeding back the change increment information to the field device. By implementing the present invention and combining the generated three-dimensional model, the remote device can mark the target device in a first-person perspective, thereby accurately guiding the field workers to perform operations, which is efficient and accurate, and realizes the remote virtual-real fusion interaction of the augmented reality method and the three-dimensional model of the power equipment.

[0071] As an optional embodiment of the present invention, Figure 5 As shown, in step S31, before receiving the data of the three-dimensional model sent by the field device, the communication method further includes:

[0072] Step S301: receiving initial video stream data of a target area sent by a field device;

[0073] Step S302: Determine the problem area based on the initial video stream data, and generate an initial key frame based on the problem area. In this embodiment, the problem area may be an area where the remote expert believes that there are problems with certain devices and wiring methods. Specifically, the remote expert may mark the problem area in text or image form, and then generate an initial key frame.

[0074] Step S303: Sending the initial key frame to the on-site device. In this embodiment, after the remote expert terminal annotates the initial video stream data sent by the on-site device, it generates an initial key frame and then sends the generated initial key frame to the on-site device.

[0075] The embodiment of the present invention provides an interactive method based on multi-feature recognition, which combines remote experts to draw and annotate video stream data collected by on-site workers from a first-person perspective, thereby generating initial key video stream segments, and can efficiently and accurately collect and store key frames of the video stream.

[0076] The following describes in detail the interaction based on multi-feature recognition of the above embodiment in combination with a specific implementation method. Specifically, the video frame is the most basic component of the video stream. The video frame with the richest information is extracted, and the main content in the video frame is converted into high-level semantic information for structured information storage. The information contained in the video stream is divided into underlying feature information, key image frame information and high-level semantic information. The underlying feature information refers to the extraction of global features, local features and structural features of the image. Global features are basic features such as shape, color, texture, etc. of the image; local features extract feature point sets of the video image for feature matching; structural features reflect the geometric and spatial-temporal relationships between image features. Key image frame information refers to the extraction of key frames based on the underlying features and target information of the image, and the information difference between frames or the information richness of the video frame is expressed by fusing multiple underlying feature information, and then representative video frames are screened out. High-level semantic information refers to the semantic logical description and feature expression based on the target and content contained in the video. Using deep learning technology, we train targeted models based on an appropriate amount of picture sets, extract target semantics, scene semantics, image semantics, etc., synthesize the extracted semantic information, and extract text sentences to logically describe the events reflected in the video, which is convenient for users to intuitively understand, store and retrieve. We combine the extracted underlying features, key image frames, and high-level semantics information for feature analysis and description, logical expression, and structured storage to achieve structured and digital storage of video streams, providing basic services for the extraction of video key frames and the recognition and matching of multiple feature points.

[0077] like Figure 6 As shown, the mobile smart terminal communicates with the background server through a wireless network. The information of multiple power equipment can be registered in the background server. The power equipment can be pre-associated with text annotations, three-dimensional models, etc. The text annotations and three-dimensional models can be pre-classified and stored in the background server, the rendering parameters of the three-dimensional model can be pre-determined, and the three-dimensional model can be pre-lightened.

[0078] The mobile smart terminal can download text annotations and three-dimensional models of multiple power equipment from the background server. After the download is completed, the three-dimensional model is rendered on the mobile smart terminal, and the virtual scene of the three-dimensional model and the actual scene of the power equipment are integrated, and then the three-dimensional model with superimposed text annotations is displayed and the corresponding power equipment is continuously tracked.

[0079] The embodiment of the present invention provides an interactive device based on multi-feature recognition, such as Figure 7 As shown, including:

[0080] The target video stream data acquisition module 41 is used to acquire the target video stream data of the target device; the detailed implementation content can refer to the relevant description of step S11 in the above method embodiment.

[0081] The calling module 42 is used to call the three-dimensional model of the target device according to the target video stream data; the detailed implementation content can refer to the relevant description of step S12 in the above method embodiment.

[0082] The data sending module 43 is used to send the data of the three-dimensional model to a remote device; the detailed implementation content can refer to the relevant description of step S13 in the above method embodiment.

[0083] The change increment information receiving module 44 is used to receive the change increment information of the three-dimensional model fed back by the remote device; the detailed implementation content can refer to the relevant description of step S14 in the above method embodiment.

[0084] An interactive device based on multi-feature recognition provided by an embodiment of the present invention includes: a target video stream data acquisition module 41, used to acquire the target video stream data of the target device; a calling module 42, used to call the three-dimensional model of the target device according to the target video stream data; a data sending module 43, used to send the data of the three-dimensional model to the remote device; a change increment information receiving module 44, used to receive the change increment information of the three-dimensional model fed back by the remote device. By implementing the present invention, combined with the three-dimensional model generated according to the target video stream data and the change increment information fed back by the remote device, the on-site equipment can obtain accurate guidance information, and realize the remote virtual-real fusion interaction of the augmented reality mode and the three-dimensional model of the power equipment.

[0085] The embodiment of the present invention provides an interactive device based on multi-feature recognition, such as Figure 8 As shown, including:

[0086] The data receiving module 51 is used to receive the data of the three-dimensional model sent by the field equipment; the detailed implementation content can refer to the relevant description of step S31 in the above method embodiment.

[0087] The three-dimensional model generation module 52 is used to generate a three-dimensional model according to the data of the three-dimensional model; the detailed implementation content can refer to the relevant description of step S32 in the above method embodiment.

[0088] The determination module 53 is used to determine the change increment information according to the three-dimensional model and the preset database; the detailed implementation content can refer to the relevant description of step S33 in the above method embodiment.

[0089] The change increment information sending module 54 is used to feed back the change increment information to the on-site device. For detailed implementation content, please refer to the relevant description of step S34 in the above method embodiment.

[0090] An interactive device based on multi-feature recognition provided by an embodiment of the present invention includes: a data receiving module 51 for receiving data of a three-dimensional model sent by a field device; a three-dimensional model generating module 52 for generating a three-dimensional model according to the data of the three-dimensional model; a determining module 53 for determining change increment information according to the three-dimensional model and a preset database; and a change increment information sending module 54 for feeding back the change increment information to the field device. By implementing the present invention and combining the generated three-dimensional model, the remote device can mark the target device in a first-person perspective, thereby accurately guiding the field operators to perform operations, which is efficient and accurate, and realizes the remote virtual-real fusion interaction of the augmented reality method and the three-dimensional model of the power equipment.

[0091] The embodiment of the present invention also provides a computer device, such as Fig. 9 As shown, the computer device may include a processor 61 and a memory 62, wherein the processor 61 and the memory 62 may be connected via a bus 60 or other means. Fig. 9 The connection via bus 60 is taken as an example.

[0092] The processor 61 may be a central processing unit (CPU). The processor 61 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0093] The memory 62 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the interactive method based on multi-feature recognition in the embodiment of the present invention. The processor 61 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 62, that is, realizing the interactive method based on multi-feature recognition in the above method embodiment.

[0094] The memory 62 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created by the processor 61, etc. In addition, the memory 62 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 62 may optionally include a memory remotely located relative to the processor 61, and these remote memories may be connected to the processor 61 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof. The one or more modules are stored in the memory 62, and when executed by the processor 61, execution such as Figure 2 as well as Figure 4 The interactive method based on multi-feature recognition in the illustrated embodiment.

[0095] For details of the above computer equipment, please refer to Figure 2 as well as Figure 4 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.

[0096] An embodiment of the present invention further provides a non-transitory computer-readable medium, which stores computer instructions. The computer instructions are used to enable a computer to execute an interactive method based on multi-feature recognition as described in any one of the above embodiments, wherein the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memories.

[0097] Obviously, the above embodiments are examples for clear explanation and are not intended to limit the implementation methods. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. An interactive method based on multi-feature recognition, characterized in that: include: Get the target video stream data of the target device; The method for obtaining target video stream data of a target device includes: collecting and sending initial video stream data of a target device in a target area; receiving an initial key frame sent by a remote device; determining a target key frame based on the initial key frame; obtaining target video stream data of the target device based on the target key frame; determining the target key frame based on the initial key frame includes: extracting color feature information, texture feature information and motion feature information from the initial video key frame; fusing the color feature information, texture feature information and motion feature information, and calculating the similarity of each initial video key frame respectively; determining candidate video key frames based on the similarity of each initial video key frame; determining the target key frame based on a preset adaptive algorithm; the preset adaptive algorithm is an adaptive hierarchical clustering algorithm, According to a preset adaptive algorithm, a target key frame is determined, specifically including: determining a candidate video key frame according to an initial video key frame, that is, a candidate key frame sequence; calculating the interaction information between salient images of each adjacent candidate key frame sequence, and calculating and determining the interaction information sequence; calculating the joint probability according to the normalized overlapping area and histogram of each adjacent image, and determining the clustering threshold according to the joint probability; arranging the interaction information sequence in descending order according to the interaction information value, and then taking the first frame as the first cluster according to the original time sequence of the candidate key frames, if the interaction information value between the subsequent two frames is less than or equal to the threshold, a new cluster is generated; on the contrary, the subsequent frames are divided into the current cluster, and the target key frame is determined, and the target key frame is an ordered cluster cluster and the frames in each cluster are also arranged in order according to the relevance of the original video content; Calling the three-dimensional model of the target device according to the target video stream data; Sending data of the three-dimensional model to a remote device; Receive the change increment information of the three-dimensional model fed back by the remote device.

2. The method according to claim 1, characterized in that Also includes: Displaying the change of the three-dimensional model according to the change increment information; The target device is controlled according to the change of the three-dimensional model.

3. The method according to claim 1, characterized in that The step of acquiring target video stream data of a target device according to the target key frame includes: Obtaining first video stream data; Identifying a first feature point in the first video stream data according to a preset optical flow method, and identifying a second feature point in the target key frame; When the similarity between the first feature point and the second feature point is greater than a preset similarity threshold, determining that the first video stream data matches the target key frame; When the first video stream data matches the target key frame, the first video stream is determined as the target video stream data of the target device.

4. The method according to claim 3, characterized in that Also includes: Determining a first center position of the target key frame according to the first feature point and a preset relative distance; Determining a second center position of the first video stream data according to the second feature point and a preset relative distance; The target video stream data of the target device is tracked and acquired according to the first center position and the second center position.

5. An interactive method based on multi-feature recognition, characterized in that: include: Receive data of a three-dimensional model sent by an on-site device; the data of the three-dimensional model is obtained by calling the target video stream data in an interactive method based on multi-feature recognition as described in claim 1; wherein obtaining the target video stream data of the target device includes: collecting and sending the initial video stream data of the target device in the target area; receiving the initial key frame sent by the remote device; determining the target key frame according to the initial key frame; obtaining the target video stream data of the target device according to the target key frame; determining the target key frame according to the initial key frame includes: extracting color feature information, texture feature information and motion feature information in the initial video key frame; fusing the color feature information, texture feature information and motion feature information, and calculating the similarity of each initial video key frame respectively; determining the candidate video key frame according to the similarity of each initial video key frame; determining the candidate video key frame according to the preset automatic Adaptive algorithm, determining target key frames; the preset adaptive algorithm is an adaptive hierarchical clustering algorithm, and the target key frames are determined according to the preset adaptive algorithm, specifically including: determining candidate video key frames according to the initial video key frames, that is, candidate key frame sequences; calculating the interaction information between the saliency images of each adjacent candidate key frame sequence, and calculating and determining the interaction information sequence; calculating the joint probability according to the normalized overlapping area and histogram of each adjacent image, and determining the clustering threshold according to the joint probability; arranging the interaction information sequence in descending order according to the interaction information value, and then taking the first frame as the first cluster according to the original time sequence of the candidate key frames, if the interaction information value between the subsequent two frames is less than or equal to the threshold, a new cluster is generated; on the contrary, the subsequent frames are divided into the current cluster, and the target key frames are determined, and the target key frames are ordered cluster clusters and the frames in each cluster are also arranged in order according to the relevance of the original video content; Generate a three-dimensional model according to the data of the three-dimensional model; Determining change increment information according to the three-dimensional model and a preset database; The change increment information is fed back to the field device.

6. The method according to claim 5, characterized in that Before receiving the data of the three-dimensional model sent by the field equipment, it also includes: Receiving initial video stream data of a target area sent by the field device; Determine a problem area according to the initial video stream data, and generate an initial key frame according to the problem area; The initial key frame is sent to the field device.

7. An interactive device based on multi-feature recognition, characterized in that: Used to perform the method according to any one of claims 1 to 4, comprising: A target video stream data acquisition module is used to acquire target video stream data of a target device; A calling module, used for calling the three-dimensional model of the target device according to the target video stream data; A data sending module, used for sending the data of the three-dimensional model to a remote device; The change increment information receiving module is used to receive the change increment information of the three-dimensional model fed back by the remote device.

8. An interactive device based on multi-feature recognition, characterized in that: Used to perform the method according to any one of claims 5-6, comprising: A data receiving module, used for receiving data of the three-dimensional model sent by the field equipment; A three-dimensional model generation module, used to generate a three-dimensional model according to the data of the three-dimensional model; A determination module, used to determine the change increment information according to the three-dimensional model and a preset database; The change increment information sending module is used to feed back the change increment information to the field device.

9. A computer device, characterized in that: include: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so that the at least one processor performs the steps of the interactive method based on multi-feature recognition as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the interactive method based on multi-feature recognition as described in any one of claims 1 to 6 are implemented.

Images