Space depth distance information determination method in oil field video identification

Abstract

The invention provides a spatial depth distance information judgment method in oil field video recognition, which comprises the following steps of: establishing semantic consistency regularization of an oil field operation scene image by acquiring point cloud dense pixel points in an oil field operation video image; information elements in the oil field operation video image are acquired, and semantic guidance-based time-space consistency regularization of the oil field operation scene image is established; and determining spatial depth distance information through distance determination in the oil field operation video image. According to the method, text embedding, image pixel features and point features are respectively extracted by using the CLIP and the 3D network, a positive sample and a negative sample are constructed, and semantic consistency regularization is applied by embedding the text corresponding to the point features; space-time consistency regularization is applied by forcing consistency between spatially and spatially coherent point features and their corresponding pixel features. And spatial depth distance information in oil field video identification is obtained by fusing a point cloud image mapping technology.

Description

Technical Field

[0001] The present invention relates to the technical field of safety management and control, and in particular to a method for determining spatial depth distance information in oilfield video recognition. Background Art

[0002] Target recognition technology has always been an important part of the field of computer vision. With the mature application of deep learning technology, many representative detection algorithms have been developed, such as R-CNN, Fast R-CNN, Faster R-CNN, R-FCN and FPN. However, the identification of illegal behaviors in oil fields involves many risks related to distance. Such as the rotation radius of people entering the groundbreaking operation, the distance limit between oxygen cylinders and acetylene cylinders, etc. This limits the recognition accuracy of the algorithm based on two-dimensional plane images for target detection and recognition, because traditional two-dimensional image recognition algorithms cannot obtain distance information in deep space. Other fields such as unmanned driving use the method of multi-camera interactive determination of distance information to solve this problem, but the camera deployment in oilfield construction scenes does not meet the equipment requirements of the corresponding fields. Summary of the invention

[0003] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended to be a general review, nor is it intended to identify key / important components or to delineate the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0004] The present invention proposes a method for determining spatial depth distance information in oilfield video recognition, aiming to solve the limitation that the prior art cannot obtain depth information of oilfield targets, so as to improve the target recognition accuracy and safety behavior management and control capabilities in oilfield construction scenarios.

[0005] The present invention proposes a method for determining spatial depth distance information in oilfield video recognition, the improvement of which is that it includes:

[0006] (1) By acquiring dense pixel points in the point cloud of oilfield operation video images, the semantic consistency regularization of oilfield operation scene images is established;

[0007] (2) Obtain information elements in oilfield operation video images and establish spatiotemporal consistency regularization of oilfield operation scene images based on semantic guidance;

[0008] (3) Determine the spatial depth distance information through distance judgment in oilfield operation video images.

[0009] Preferably, the step (1) comprises

[0010] (1-2) Establishing dense pixel correspondence in oilfield operation video images, and transmitting corresponding target information in the image to the point cloud through pixel pairs;

[0011] (1-2) Convert pixel-text pairs to point-text pairs Use text semantics to select positive and negative point samples for contrastive learning, and use the loss function formula L S_info Establish semantic consistency regularization;

[0012] (1-3) Align the dense pixel pairs in the point cloud information with the text to obtain the dense pixel points of the point cloud in the oilfield operation video image.

[0013] Further, the step (1-1) includes calibrating the LiDAR point cloud using corresponding images captured by six cameras.

[0014] Further, the use of corresponding images captured by six cameras to calibrate the LiDAR point cloud includes obtaining dense pixel point correspondences Among them, x i and p i denote the i-th paired image feature and point feature extracted by the image encoder and 3D network of CLIP, respectively, and M is the number of pairs.

[0015] Preferably, the step (1-2) comprises generating dense pixel-text pairs by using the MaskCLIP algorithm Use text semantics to select positive and negative samples for contrastive learning, and use the loss function formula L S_info to process;

[0016] Among them, t i It is the text embedding generated by CLIP's text encoder, which converts pixel-text pairs into point-text pairs.

[0017] Furthermore, the loss function formula L S_info as follows:

[0018]

[0019] Among them, t i ∈c represents t i Generated by the c-th class name, C is the number of classes, D represents the scalar multiplication operation, and τ is the temperature parameter τ>0.

[0020] Preferably, the step (2) comprises

[0021] (2-1) Given an image I and a temporally continuous LiDAR point cloud Where K is the number of scans in S seconds;

[0022] (2-2) The image and the first frame of the point cloud P1 are matched by pixel-point pairing. Make a match;

[0023] (2-3) Align the remaining point clouds with the first frame through the calibration matrix and map them to the image;

[0024] (2-4) All pixel-point-text pairs can be obtained in a short time

[0025] (2-5) Divide the entire spliced ​​point cloud into regular grids Among them, points that are consecutive in time are located in the same grid;

[0026] (2-6) Applying spatiotemporal consistency constraints within each grid:

[0027]

[0028] Among them, L SSR is the loss function, Represents pixel pairs Located in the nth grid, It is a cross-modal fusion feature guided by semantics, and its specific formula is:

[0029]

[0030] in and The attention weight is calculated by the following formula, and λ is the temperature parameter:

[0031]

[0032]

[0033] (2-7) Align the image and text to obtain the information elements in the oilfield operation video image.

[0034] Preferably, the step (3) comprises

[0035] (3-1) According to the loss function L in step (1-2) S_info And the loss function L of step (2-6) SSR Perform end-to-end network training to achieve spatial alignment between LiDAR point clouds and two-dimensional information points.

[0036] (3-2) For the distance calculation between two targets t1 and t2 in video recognition, the pixel-point-text pair Obtain the corresponding point clouds p1 and p2, and return the spatial distance between the target objects t1 and t2 through the distance value between the point clouds;

[0037] (3-3) The distance determination in the oilfield operation video image is converted into the corresponding distance in the point cloud.

[0038] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0039] The present invention obtains spatial depth distance information in oilfield video recognition by integrating point cloud image mapping technology, and has a good accuracy improvement rate in the recognition mode of distance-related illegal behaviors such as people standing within the radius of the excavator arm, people standing within the radius of the crane, abnormal spacing between hot equipment, upper and lower cross operations, and people standing on the oil pipe row.

[0040] The present invention converts pixel-text pairs into point-text pairs The text semantics is used to select positive and negative samples for contrastive learning, and a new loss function formula L is proposed. S_info , so that the points with the same semantics in the present invention will be restricted to be close to the same text embedding, and the points with different semantics will be pushed away, thus solving the shortcomings of the existing point cloud-pixel cross-modal. S_info and L SSR Combined, the entire network is trained end-to-end, and a self-training method is further proposed to randomly switch between paired image pixel labels and their own predicted point labels. Satisfactory performance can be achieved by training the 3D network only with its own predictions. This switchable self-training strategy increases the number of positive and negative samples by switching point pseudo labels, which is more conducive to cross-modal knowledge distillation extraction.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0043] Figure 1 The present invention is a flow chart of a method for determining spatial depth distance information in oilfield video recognition according to an exemplary embodiment.

[0044] Figure 2 2 is a flowchart of an image pixel-to-point mapping diagram (left) and a semantic-guided fusion feature generation (right) according to an exemplary embodiment. DETAILED DESCRIPTION

[0045] The following description and accompanying drawings fully illustrate specific embodiments of the present invention so that those skilled in the art can practice them. The examples represent possible variations only. Unless explicitly required, separate components and functions are optional, and the order of operations can vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of the present invention includes the entire scope of the claims, and all available equivalents of the claims. In this article, each embodiment may be represented individually or generally by the term "invention", which is only for convenience, and if more than one invention is in fact disclosed, it is not intended to automatically limit the scope of the application to any single invention or inventive concept. In this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, without requiring or implying any actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. The various embodiments are described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. As for the structures, products, etc. disclosed in the embodiments, since they correspond to the parts disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0047] like Figure 1 As shown in the figure, in the process of identifying the depth information of the target in the oilfield construction scene, the present invention proposes a method for determining the spatial depth distance information in oilfield video recognition. The method first uses CLIP and 3D network to extract text embedding, image pixel features and point features respectively, then constructs positive and negative samples based on the knowledge of CLIP, and finally applies semantic consistency regularization by pulling the point features onto their corresponding text embeddings. At the same time, the method applies spatiotemporal consistency regularization by enforcing the consistency between spatiotemporally coherent point features and their corresponding pixel features.

[0048] The purpose of the present invention is to provide a method for determining spatial depth distance information in oilfield video recognition. To achieve the above purpose, the present invention provides the following technical solutions to identify the distance when a person steps under the boom in an oilfield hoisting operation scene:

[0049] Step 1: By acquiring dense pixel points of the point cloud in the oilfield operation video image, the semantic consistency regularization of the oilfield operation scene image is established;

[0050] 1) Establish dense pixel correspondence in the oilfield operation video image, and transfer the corresponding target information in the image to the 3D point cloud through pixel pairs. Specifically, this method uses the corresponding images captured by six cameras to calibrate the LiDAR point cloud. Therefore, dense pixel correspondence can be obtained accordingly. where x i and p i denote the i-th paired image feature and point feature extracted by the image encoder and 3D network of CLIP, respectively, and M is the number of pairs.

[0051] 2) Establish a semantic consistency regularization method by using the MaskCLIP algorithm to generate dense pixel-text pairs where t i is the text embedding generated by CLIP's text encoder. Then the pixel-text pairs are converted to point-text pairs And use text semantics to select positive and negative point samples for contrastive learning. The innovative loss function used in the present invention is as follows:

[0052]

[0053] Among them, t i ∈c represents t i Generated by the c-th class name, C is the number of classes, D represents the scalar multiplication operation, and τ is the temperature parameter τ>0. Since the text consists of class names placed in a predefined template, the text embedding represents the semantic information of the corresponding class. In the method of the present invention, points with the same semantics will be constrained to be close to the same text embedding, and points with different semantics will be pushed away.

[0054] 3) The dense pixel pairs in the information on the point cloud are aligned with the text to obtain the dense pixel points of the point cloud of the operator and the crane arm.

[0055] Step 2: Obtain information elements in the oilfield operation video image and establish the spatiotemporal consistency regularization of the oilfield operation scene image based on semantic guidance;

[0056] After the semantic consistency regularization of the oilfield operation scene image, the present invention alleviates this problem by imposing soft constraints on points in local space and time based on semantically guided spatiotemporal consistency regularization. The specific implementation method is as follows:

[0057] 2-1) If Figure 2 As shown, given an image I and a temporally continuous LiDAR point cloud Where K is the number of scans in S seconds.

[0058] 2-2) Compare the image and the first frame of point cloud P1 through pixel-point pairing to make a match.

[0059] 2-3) Align the rest of the point clouds with the first frame via the calibration matrix and map them onto the image.

[0060] 2-4) Then all pixel-point-text pairs can be obtained in a short time

[0061] 2-5) Then divide the entire spliced ​​point cloud into regular grids Wherein temporally consecutive points are located in the same grid.

[0062] 2-6) Then, the temporal and spatial consistency constraints are imposed in each grid through the objective function proposed by the present invention:

[0063]

[0064] Among them, L SSR is the loss function, Represents pixel pairs Located in the nth grid, It is a cross-modal fusion feature guided by semantics, and its specific formula is:

[0065]

[0066] in and The attention weight is calculated by the following formula, and λ is the temperature parameter:

[0067]

[0068]

[0069] 2-7) The image and the text are thereby aligned to obtain information elements of the operator and the crane arm in the image.

[0070] Step 3: Determine the spatial depth distance information through distance determination in the oilfield operation video image;

[0071] 3-1) Combine the loss function L in step 1-2 S_info and the loss function L of steps 2-6 SSR The entire network is trained end-to-end, and finally the spatial alignment of the Lidar point cloud and the two-dimensional information points is formed.

[0072] 3-2) For the distance calculation between two targets t1 and t2 in video recognition, the pixel-point-text pair The corresponding point clouds p1 and p2 are obtained, and the spatial distance between the target objects t1 and t2 is obtained by returning the distance value between the point clouds. Specifically, the distance between the personnel and the boom is determined and converted into the corresponding distance on the point cloud.

[0073] The present invention obtains spatial depth distance information in oilfield video recognition by integrating point cloud image mapping technology. At the same time, it also plays a great role in improving the accuracy of distance-related illegal behavior recognition modes such as people standing within the radius of the excavator arm, people standing within the radius of the crane, abnormal spacing between hot equipment, upper and lower cross operations, and people standing on the oil pipe row.

[0074] Name of violation Video recognition accuracy The recognition accuracy of the present invention People standing within the radius of the digging arm 87.24％ 96.87％ People must stand within the radius of the crane 84.31％ 95.43％ Abnormal spacing between hot work equipment 74.2％ 97.32％ Cross-operation 57.98％ 72.48％

[0075]

[0076] It should be understood that the above scheme has presented the foregoing description of specific exemplary embodiments of the present invention for the purpose of illustration and description. It is not intended to exclude or limit the present invention to the precise form disclosed, and it is obvious that many modifications and changes are feasible in view of the above teachings. Select and describe exemplary embodiments to explain certain principles of the present invention and their practical application, so that other technical personnel in the field can make or use various exemplary embodiments of the present invention, and various alternatives and modifications thereof. Its purpose is that the scope of the present invention will be defined by the claims attached to the present invention and their equivalents.

[0077] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for determining spatial depth distance information in oilfield video recognition, characterized in that: include: (1) By acquiring dense pixel points in the point cloud of oilfield operation video images, the semantic consistency regularization of oilfield operation scene images is established; (2) Obtain information elements in oilfield operation video images and establish spatiotemporal consistency regularization of oilfield operation scene images based on semantic guidance; (3) Determine the spatial depth distance information through distance judgment in oilfield operation video images.

2. The method for determining spatial depth distance information in oilfield video recognition according to claim 1, characterized in that: The step (1) comprises (1-1) Establishing dense pixel correspondence in oilfield operation video images, and transmitting corresponding target information in the image to the point cloud through pixel pairs; (1-2) Convert pixel-text pairs to point-text pairs Use text semantics to select positive and negative point samples for contrastive learning, and use the loss function formula L S_info Establish semantic consistency regularization; (1-3) Align the dense pixel pairs in the point cloud information with the text to obtain the dense pixel points of the point cloud in the oilfield operation video image.

3. The method for determining spatial depth distance information in oilfield video recognition according to claim 2 is characterized in that: The step (1-1) includes calibrating the LiDAR point cloud using corresponding images captured by six cameras.

4. The method for determining spatial depth distance information in oilfield video recognition according to claim 3 is characterized in that: The method of calibrating the LiDAR point cloud using corresponding images captured by six cameras includes obtaining dense pixel point correspondences. Among them, x i and p i denote the i-th paired image feature and point feature extracted by the image encoder and 3D network of CLIP, respectively, and M is the number of pairs.

5. The method for determining spatial depth distance information in oilfield video recognition according to claim 2, characterized in that: The step (1-2) comprises generating dense pixel-text pairs by using the MaskCLIP algorithm Use text semantics to select positive and negative samples for contrastive learning, and use the loss function formula L S_info to process; Among them, t i It is the text embedding generated by CLIP's text encoder, which converts pixel-text pairs into point-text pairs.

6. The method for determining spatial depth distance information in oilfield video recognition according to claim 5, characterized in that: The loss function formula L S_info as follows: Among them, t i ∈c represents t i Generated by the c-th class name, C is the number of classes, D represents the scalar multiplication operation, and τ is the temperature parameter τ>0.

7. The method for determining spatial depth distance information in oilfield video recognition according to claim 1, characterized in that: The step (2) comprises (2-1) Given an image I and a temporally continuous LiDAR point cloud Where K is the number of scans in S seconds; (2-2) The image and the first frame of the point cloud P1 are matched by pixel-point pairing. Make a match; (2-3) Align the remaining point cloud with the first frame through the calibration matrix and map it to the image; (2-4) All pixel-point-text pairs can be obtained in a short time (2-5) Divide the entire spliced ​​point cloud into regular grids Among them, points that are consecutive in time are located in the same grid; (2-6) Apply spatiotemporal consistency constraints within each grid: Among them, L SSR is the loss function, Represents pixel pairs Located in the nth grid, It is a cross-modal fusion feature guided by semantics, and its specific formula is: in and The attention weight is calculated by the following formula, and λ is the temperature parameter: (2-7) Align the image and text to obtain the information elements in the oilfield operation video image.

8. The method for determining spatial depth distance information in oilfield video recognition according to claim 1, characterized in that: The step (3) comprises (3-1) According to the loss function L in step (1-2) S_info And the loss function L of step (2-6) SSR Perform end-to-end network training to form spatial alignment between LiDAR point cloud and two-dimensional information points; (3-2) For the distance calculation between two targets t1 and t2 in video recognition, the pixel-point-text pair Obtain the corresponding point clouds p1 and p2, and return the distance between the point clouds to obtain the spatial distance between the target objects t1 and t2; (3-3) The distance determination in the oilfield operation video image is converted into the corresponding distance in the point cloud.

Images