A target identification method, system and edge computing device for tunnel

Abstract

The present invention provides a method, system and edge computing device for target recognition in a tunnel. The target recognition system includes a network camera, a network switching device, and an edge computing device. The network camera and the edge computing device are connected to the network switching device via an Ethernet cable, so that the edge computing device can access the network camera through a network address. The network camera includes a front camera and a rear camera. The beneficial effects of the present invention are: the present invention introduces some deep learning ideas on the basis of traditional machine learning, combines the unstable lighting in the tunnel, the strong vibration of the rail locomotive, etc., and designs a stable target recognition method. This method can be deployed on the edge computing device, the deployment cost is reduced, and the real-time performance and accuracy are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a target recognition method, system and edge computing device for use in a tunnel. Background Art

[0002] With the development of industrial automation and the advancement of artificial intelligence technology, many production scenarios require the help of artificial intelligence systems to complete judgments, which can make up for the inaccuracy and lag of manual judgments to a certain extent. However, the current integration and intelligence of artificial intelligence systems are not high enough. In the face of the diversity of industrial scenarios, simply using traditional target recognition algorithms such as the RCNN series and Faster-RCNN series cannot achieve satisfactory results in both accuracy and speed, resulting in the inability to accurately detect and locate some important targets in the case of changing scenes in tunnels. Summary of the invention

[0003] The present invention provides a method for identifying a target in a tunnel, comprising the following steps:

[0004] Step 1: Read the video data from the front camera and the rear camera;

[0005] Step 2: Combine the video data of the front camera and the rear camera into an array. When the reading is successful, add the current frame to the array.

[0006] Step 3: Preprocess the current frame images stored in the array by the front camera and the rear camera. When the current frame image is judged to have unsatisfactory lighting, skip this judgment cycle; when the current frame image is judged to have ideal lighting, scale or expand the image size to a set size, and quantize the pixel value of each channel of each pixel;

[0007] Step 4: Flatten the image into a one-dimensional array by row and input it into the neural network model to obtain the predicted result, which includes the type of each classification and its coordinates in the image;

[0008] Step 5: Perform non-maximum suppression on the predicted results, filter the repeated detection results, remove redundant detection results, and retain the one closest to the true value;

[0009] Step 6: Mark the targets in the result with rectangular boxes in the image according to their classification and coordinates, and mark the labels on the rectangular boxes in the form of text. The content of the labels includes the classification category and confidence score; at the same time, the results are communicated to other modules for use.

[0010] As a further improvement of the present invention, in step 1, the video data of the front camera and the rear camera are read simultaneously in a multi-threaded manner.

[0011] As a further improvement of the present invention, the step 1 comprises:

[0012] Step A: Determine whether the read video data is encapsulated using the RTSP protocol. If so, execute step B. Otherwise, exit the program and give an error warning.

[0013] Step B: Initialize the program's running log and device parameters. If the GPU of the edge computing device works properly, use the GPU to execute the calculation part of the program, and the rest of the program is executed by the CPU. If the GPU is not detected to be working properly, use the CPU to execute the calculation part of the program, and the rest of the program is also executed by the CPU. And load the neural network model.

[0014] As a further improvement of the present invention, in step 2, the video data of the front camera and the rear camera are read using OpenCV and merged into an array; when the reading is successful, the current frame is added to the array; when the network is unstable, the read image data is of type None, and the blank frame is added to the array.

[0015] As a further improvement of the present invention, in step 3, the current frame images stored in the array by the front camera and the rear camera are preprocessed. When it is found that the average value of the image pixels is close to the black or white image, it is determined that the lighting is not ideal and the current judgment cycle is skipped. When it is determined that the lighting is ideal, the image size is scaled or expanded to 640*640, and the pixel value of each channel of each pixel is quantized and standardized from 0 to 255 to 0 to 1.

[0016] As a further improvement of the present invention, in step 6, the result is communicated to other modules for use in a JSON encoding manner.

[0017] As a further improvement of the present invention, in step 4, in the neural network model, the preprocessed data is convolved to obtain a feature image of the original image, and the input and output are added on the channel using a residual network. The obtained feature image is smaller in size than the original image, but the number of channels is increased; each pixel of the obtained feature image corresponds to the features of multiple pixels at the corresponding position of the original image. When the feature map indicates that a target for target recognition exists at a certain pixel position, it indicates that a target for target recognition exists at the corresponding position on the original image.

[0018] As a further improvement of the present invention, in the neural network model, the original image is convolved with convolution kernels of different sizes to obtain feature maps at three different scales, and the contents identified by the feature maps at different scales are fused to obtain the target recognition content of the entire original image. The fused new feature map includes: the coordinates of the identified target, a rectangular box is used to describe the coordinates of the target, and the description method is to take the upper left corner vertex of the original image as the origin, increase horizontally to the right in the X direction, and increase vertically downward in the Y direction, and record the coordinates of the upper left corner and the lower right corner of the rectangular box. The center point of the identified target should be in the middle of the upper left corner coordinate and the lower right corner coordinate, and the coordinates are given in percentage form; the new feature map also includes the confidence score of the identified target, and the new feature map also includes the category of the identified target.

[0019] The present invention also provides an edge computing device, comprising: a memory, a processor, and a computer program stored on the memory, wherein the computer program is configured to implement the steps of a target identification method when called by the processor.

[0020] The present invention also provides a target recognition system for use in a tunnel, comprising a network camera, a network switching device, and an edge computing device, wherein the network camera and the edge computing device are connected to the network switching device via an Ethernet cable, so that the edge computing device can access the network camera via a network address, and the network camera comprises a front camera and a rear camera.

[0021] As a further improvement of the present invention, the network camera has a focal length of 4 mm, a resolution of 1280*720P, a frame rate of 25, an upper limit of a bit rate of 8192Kbps, and H.264 video encoding. The edge computing device is equipped with a Linux system and a GTX 1660 graphics card. The front camera is installed at the head of the locomotive, and the rear camera is installed at the rear of the locomotive. The front camera installed at the head of the locomotive is located above the window directly in front of the locomotive, and the field of view of the front camera covers the track area directly in front of the locomotive; the rear camera installed at the rear of the locomotive is located below the car plate of the last segment car, and its installation angle is to look directly at the rear direction of the car tail, and the field of view must cover the track area directly behind the locomotive; the edge computing device and the network switching device are respectively deployed in the equipment chassis in the locomotive cab.

[0022] The beneficial effects of the present invention are as follows: the present invention introduces some deep learning ideas based on traditional machine learning, and designs a stable target recognition method in combination with unstable lighting in tunnels, strong vibration of rail locomotives, etc. This method can be deployed on edge computing devices, with reduced deployment costs and significantly improved real-time performance and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0024] The present invention discloses a target recognition system for use in a tunnel, including a hardware architecture and a software code, wherein the software code runs on the basis of the hardware architecture. In order to realize the function of target recognition, the hardware architecture of the present invention should include the following hardware devices: two network cameras with a focal length of 4 mm, a resolution of 1280*720P, a frame rate of 25, a bit rate upper limit of 8192Kbps, and H.264 video encoding, an edge computing device equipped with a Linux system and a GTX 1660 graphics card, and a network switching device. The network camera and the edge computing device are connected to the network switching device via an Ethernet cable, so that the edge computing device can access a specific network camera through a network address.

[0025] The two network cameras should be installed as follows: one is installed on the head of the locomotive (called the front camera), and the other is installed on the rear of the locomotive (called the rear camera). The front camera installed on the head of the locomotive is located above the window directly in front of the locomotive. Its installation angle is to look directly in front of the front of the locomotive and tilt downward at an angle of about 15 degrees. The field of view must cover the track area directly in front of the locomotive; the rear camera installed on the rear of the locomotive is located below the car plate of the last segment car. Its installation angle is to look directly behind the rear of the car, and the field of view must cover the track area directly behind the locomotive. The distance between the front and rear cameras is the train length of the entire vehicle, which is about 50 meters. Edge computing equipment and network switching equipment should be deployed in the equipment chassis in the locomotive cab respectively, and maintain good ventilation and heat dissipation.

[0026] The software code of the present invention runs on the edge computing device in the above-mentioned hardware architecture, using the video stream data of the front and rear cameras as input signals. After network exchange, the target recognition software code module is run, and the results are output in the form of byte streams and sent to other software modules for their use.

[0027] like Figure 1 As shown, the present invention discloses a target identification method for a tunnel, comprising the following steps:

[0028] Step 1: Read the video data of the front camera and the rear camera; in step 1, the video data of the front camera and the rear camera are read simultaneously in a multi-threaded manner.

[0029] In step 1, after the front camera and the rear camera capture the video data, they are encapsulated through the RTSP protocol to provide streaming media services for access by other network devices. The video data can also be encapsulated using HTTP, RTMP, or WebRTC.

[0030] The step 1 comprises:

[0031] Step A: Determine whether the read video data is encapsulated using the RTSP protocol. If so, execute step B. Otherwise, exit the program and give an error warning.

[0032] Step B: Initialize the program's running log and device parameters. If the GPU of the edge computing device works properly, use the GPU to execute the calculation part of the program, and the rest of the program is executed by the CPU. If the GPU is not detected to be working properly, use the CPU to execute the calculation part of the program, and the rest of the program is also executed by the CPU. And load the neural network model.

[0033] Step 2: Merge the video data of the front camera and the rear camera into an array, and when the reading is successful, add the current frame into the array; as a preferred embodiment of step 2, the video data of the front camera and the rear camera are read using OpenCV and merged into an array; when the reading is successful, add the current frame into the array; when the network is unstable, the image data read is of type None. At this time, in order to ensure the normal operation of the program, add the blank frame into the array to ensure that the subsequent network model can read the data.

[0034] Step 3: Preprocess the current frame image stored in the array by the front camera and the rear camera. When the current frame image is determined to be under ideal lighting, skip this judgment cycle. When the current frame image is determined to be under ideal lighting, scale or expand the image size to a set size, and quantize the pixel value of each channel of each pixel. As a preferred embodiment of step 3, preprocess the current frame image stored in the array by the front camera and the rear camera. When it is found that the average value of the image pixels is close to a black or white image, it is determined to be under ideal lighting, and skip this judgment cycle. When it is determined to be under ideal lighting, scale or expand the image size to 640*640, and quantize the pixel value of each channel of each pixel, and standardize it from 0 to 255 to 0 to 1.

[0035] Step 4: Flatten the image into a one-dimensional array by row and input it into the neural network model to obtain the predicted result, which includes the type of each classification and its coordinates in the image;

[0036] Step 5: Perform non-maximum suppression on the predicted results, filter the repeated detection results, remove redundant detection results, and retain the one closest to the true value;

[0037] Step 6: Mark the targets in the result with rectangular boxes in the image according to their classification and coordinates, and mark the labels on the rectangular boxes in the form of text. The content of the label includes the classification category and confidence score; at the same time, the results are communicated to other modules through JSON encoding.

[0038] In the neural network model, the preprocessed data is convolved to obtain the feature image of the original image, and the residual network is used to add the input and output on the channel, so that the parameters in the training process will not be unable to be trained due to gradient disappearance or gradient explosion. The acquired feature image is smaller in size than the original image, but the number of channels is increased; each pixel of the acquired feature image corresponds to the features of multiple pixels at the corresponding position of the original image. For example, if the original image is 14*14 and the feature image is 7*7, then each pixel value of the feature image corresponds to 2*2 of the original image, a total of 4 pixels. When the feature map indicates that a target for target recognition exists at a certain pixel position, it means that a target for target recognition exists at the corresponding position on the original image.

[0039] In the neural network model, targets of different sizes in the original image can be found through feature maps of different scales, and the feature maps are fused into new feature maps according to the channel direction. The present invention performs convolution operations on the original image through convolution kernels of different sizes to obtain feature maps at three different scales, fuses the contents identified by the feature maps at different scales, and obtains the target recognition content of the entire original image. The fused new feature map contains the following contents: the coordinates of the identified target, and a rectangular box is used to describe the coordinates of the target. The description method is to use the upper left corner vertex of the original image as the origin, increase in the X direction horizontally to the right, and increase in the Y direction vertically downward, and record the coordinates of the upper left corner and the lower right corner of the rectangular box. The center point of the identified target should be between the upper left corner coordinate and the lower right corner coordinate. Considering the different sizes of the input images, the coordinates are given in percentage form; the new feature map also includes the confidence score of the identified target, that is, the probability that the target is true. In this algorithm, the result with a confidence score greater than 0.4 (which can also be set to other 0-1 values ​​in other usage scenarios) is considered to be true. The new feature map also includes the category of the identified target, given in the form of an integer, such as 1 for category 1, 2 for category 2, and so on.

[0040] The output result is transmitted to the IO module in the form of byte stream in JSON encoding. The specific content includes: protocol ID, whether the target is recognized, image size length, image size width, number of recognized targets, and recognized target coordinates. Other modules need to decode the byte stream first and then decode it in JSON encoding to obtain specific data. Of course, the output result can also be given in other formats, such as XML, string.

[0041] The present invention solves the target recognition method required for intelligent construction in a tunnel, and can recognize the category of a desired object and its coordinates in a diagram according to requirements.

[0042] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A target identification method for a tunnel, characterized in that: The steps include: Step 1: Read the video data from the front camera and the rear camera; Step 2: Combine the video data of the front camera and the rear camera into an array. When the reading is successful, add the current frame to the array. Step 3: Preprocess the current frame images stored in the array by the front camera and the rear camera. When the current frame image is judged to have unsatisfactory lighting, skip this judgment cycle; when the current frame image is judged to have ideal lighting, scale or expand the image size to a set size, and quantize the pixel value of each channel of each pixel; Step 4: Flatten the image into a one-dimensional array by row and input it into the neural network model to obtain the predicted result, which includes the type of each classification and its coordinates in the image; Step 5: Perform non-maximum suppression on the predicted results, filter the repeated detection results, remove redundant detection results, and retain the one closest to the true value; Step 6: Mark the targets in the result with rectangular boxes in the image according to their classification and coordinates, and mark the labels on the rectangular boxes in the form of text. The content of the labels includes the classification category and confidence score; at the same time, the results are communicated to other modules for use.

2. The target recognition method according to claim 1, characterized in that: In step 1, the video data of the front camera and the rear camera are read simultaneously in a multi-threaded manner.

3. The target recognition method according to claim 1, characterized in that: The step 1 comprises: step A: determining whether the read video data is encapsulated using the RTSP protocol, if so, executing step B, otherwise, exiting the program and giving an error warning; Step B: Initialize the program's running log and device parameters. If the GPU of the edge computing device works properly, use the GPU to execute the calculation part of the program, and the rest of the program is executed by the CPU. If the GPU is not detected to be working properly, use the CPU to execute the calculation part of the program, and the rest of the program is also executed by the CPU. And load the neural network model.

4. The target recognition method according to claim 1, characterized in that: In step 2, the video data of the front camera and the rear camera are read using OpenCV and merged into an array; when the reading is successful, the current frame is added to the array; When the network is unstable, the image data read is of type None, and blank frames are added to the array.

5. The target recognition method according to claim 1, characterized in that: In step 3, the current frame images stored in the array by the front camera and the rear camera are preprocessed. When it is found that the average value of the image pixels is close to the black or white image, it is determined that the lighting is not ideal and the current judgment cycle is skipped. When it is determined that the lighting is ideal, the image size is scaled or expanded to 640*640, and the pixel value of each channel of each pixel is quantized and standardized from 0 to 255 to 0 to 1.

6. The target recognition method according to claim 1, characterized in that: In step 6, the result is communicated to other modules through JSON encoding.

7. The target recognition method according to claim 1, characterized in that: In step 4, in the neural network model, the preprocessed data is convolved to obtain the feature image of the original image, and the input and output are added on the channel using a residual network. The obtained feature image is smaller in size than the original image, but the number of channels is increased; each pixel of the obtained feature image corresponds to the features of multiple pixels at the corresponding position of the original image. When the feature map indicates that there is a target for target recognition at a certain pixel position, it indicates that there is a target for target recognition at the corresponding position on the original image.

8. The target recognition method according to claim 7, characterized in that: In the neural network model, the original image is convolved with convolution kernels of different sizes to obtain feature maps at three different scales. The contents recognized by the feature maps at different scales are fused to obtain the target recognition content of the entire original image. The fused new feature map contains the following contents: the coordinates of the recognized target. A rectangular box is used to describe the coordinates of the target. The description method is to take the upper left corner vertex of the original image as the origin, increase horizontally to the right in the X direction, and increase vertically downward in the Y direction. The coordinates of the upper left corner and the lower right corner of the rectangular box are recorded. The center point of the recognized target should be in the middle of the upper left corner coordinate and the lower right corner coordinate, and the coordinates are given in percentage form; the new feature map also includes the confidence score of the recognized target, and the new feature map also includes the category of the recognized target.

9. An edge computing device, characterized in that: include: A memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the target recognition method according to any one of claims 1 to 8 when called by the processor.

10. A target recognition system for use in a tunnel, characterized in that: It includes a network camera, a network switching device, and the edge computing device as described in claim 9, wherein the network camera and the edge computing device are connected to the network switching device via an Ethernet cable, so that the edge computing device can access the network camera via a network address, and the network camera includes a front camera and a rear camera.

11. The target recognition system according to claim 10, characterized in that: The network camera has a focal length of 4 mm, a resolution of 1280*720P, a frame rate of 25, an upper limit of 8192Kbps bit rate, and H.264 video encoding. The edge computing device is equipped with a Linux system and a GTX 1660 graphics card. The front camera is installed on the head of the locomotive, and the rear camera is installed on the rear of the locomotive. The front camera installed on the head of the locomotive is located above the window directly in front of the locomotive, and the field of view of the front camera covers the track area directly in front of the locomotive; the rear camera installed at the rear of the locomotive is located below the car plate of the last segment car, and its installation angle is to look directly behind the rear of the car, and the field of view must cover the track area directly behind the locomotive; the edge computing device and the network switching device are respectively deployed in the equipment chassis in the locomotive cab.

Images