Foreign vehicle monitoring method, device and equipment and storage medium

Abstract

The invention discloses a foreign vehicle monitoring method, device and equipment and a storage medium, which are used for quickly detecting foreign vehicles in a set area and positioning and monitoring the foreign vehicles. The method is applied to an edge server, and comprises the following steps: receiving images sent by a plurality of image acquisition devices located in a set area; detecting the plurality of images by adopting a pre-trained detection model, and determining at least one image containing the foreign vehicle; determining vehicle information of the foreign vehicle according to the at least one image, and determining position information of the foreign vehicle according to the position of an image acquisition device for acquiring the at least one image; and the vehicle information and the position information of the foreign vehicle are sent to the data center, so that the data center monitors the moving track of the foreign vehicle in the set area.

Description

Technical Field

[0001] The present application relates to the field of wireless communication technology, and more particularly to a method, device, equipment and storage medium for monitoring an incoming vehicle. Background Art

[0002] In order to ensure the safety of people and property in the park or a specific area, it is necessary to promptly identify foreign vehicles entering the park, and to locate and monitor foreign vehicles in real time. There are two main traditional methods for identifying vehicles, one is executed by the camera chip, and the other is executed by the cloud computing platform. Among them, the solution executed by the camera chip requires the chip to have high computing power and high cost. The solution executed by the cloud requires the camera to upload the real-time collected images to the cloud for identification. This method has a high occupancy rate of transmission bandwidth and cannot work well in areas with no signal or limited signal. Summary of the invention

[0003] The present application provides a method and device for monitoring an incoming vehicle, which are used to quickly detect incoming vehicles in a set area and perform positioning monitoring on the incoming vehicles.

[0004] In a first aspect, the present application provides a method for monitoring an external vehicle, which is applied to an edge server and includes:

[0005] receiving images sent by a plurality of image acquisition devices located in a set area;

[0006] Using a pre-trained detection model to detect multiple images, and determining at least one image containing a foreign vehicle;

[0007] determining vehicle information of the foreign vehicle based on the at least one image, and determining position information of the foreign vehicle based on the position of an image acquisition device that acquires the at least one image;

[0008] The vehicle information and location information of the external vehicle are sent to the data center, so that the data center can monitor the movement track of the external vehicle in the set area.

[0009] In one or more embodiments, the plurality of image acquisition devices include image acquisition devices deployed at fixed positions within the set area, and image acquisition devices deployed in an interior vehicle located within the set area.

[0010] In one or more embodiments, when the plurality of image acquisition devices are image acquisition devices deployed inside a vehicle, the method further includes:

[0011] receiving vehicle information and position information of the external vehicle sent by the multiple image acquisition devices; the vehicle information of the external vehicle sent by the multiple image acquisition devices is obtained by the multiple image acquisition devices detecting the collected images based on the detection model, and the position information of the external vehicle sent by the multiple image acquisition devices is determined by the multiple image acquisition devices according to their own positions;

[0012] The received vehicle information and location information of the external vehicle are forwarded to the data center.

[0013] In one or more embodiments, before receiving images sent by multiple image acquisition devices located in a set area, the method further includes:

[0014] Receiving the detection model sent by the cloud computing platform;

[0015] After determining the vehicle information and location information of the foreign vehicle, the method further includes:

[0016] The at least one image and a tag of the at least one image are sent to the cloud computing platform for the cloud computing platform to update the detection model; the tag of the at least one image is used to indicate that the foreign vehicle is included in the at least one image.

[0017] In one or more embodiments, the sending the vehicle information and location information of the external vehicle to the data center server includes:

[0018] Establishing a satellite communication link with the data center through satellite network side equipment;

[0019] The vehicle information and location information of the external vehicle are sent to the data center via the satellite communication link.

[0020] In one or more embodiments, the detection model is trained in the following manner:

[0021] Acquire a training sample set; the training sample set includes a plurality of labeled sample images;

[0022] Extracting a set number of sample images from the training sample set and combining them to obtain new sample images, and using the new sample images to expand the training sample set;

[0023] The detection model is trained using the expanded training sample set.

[0024] In a second aspect, the present application proposes a monitoring device for a foreign vehicle, the device being an edge server, or the device being applied to the edge server, the device comprising:

[0025] A communication unit, used for receiving images sent by a plurality of image acquisition devices located in a set area;

[0026] A processing unit, configured to detect the plurality of images using a pre-trained detection model, and determine at least one image containing a foreign vehicle;

[0027] The processing unit is further configured to determine the vehicle information of the foreign vehicle based on the at least one image, and determine the position information of the foreign vehicle based on the position of an image acquisition device that acquires the at least one image;

[0028] The communication unit is also used to send the vehicle information and location information of the external vehicle to the data center, so that the data center can monitor the movement trajectory of the external vehicle in the set area.

[0029] In one or more embodiments, the plurality of image acquisition devices include image acquisition devices deployed at fixed positions within the set area, and image acquisition devices deployed in an interior vehicle located within the set area.

[0030] In one or more embodiments, when the plurality of image acquisition devices are image acquisition devices deployed inside a vehicle, the communication unit is further configured to:

[0031] receiving vehicle information and position information of the external vehicle sent by the multiple image acquisition devices; the vehicle information of the external vehicle sent by the multiple image acquisition devices is obtained by the multiple image acquisition devices detecting the collected images based on the detection model, and the position information of the external vehicle sent by the multiple image acquisition devices is determined by the multiple image acquisition devices according to their own positions;

[0032] The received vehicle information and location information of the external vehicle are forwarded to the data center.

[0033] In one or more embodiments, before receiving images sent by multiple image acquisition devices located in a set area, the communication unit is further used to:

[0034] Receiving the detection model sent by the cloud computing platform;

[0035] After determining the vehicle information and location information of the foreign vehicle, the communication unit is further used to:

[0036] The at least one image and a tag of the at least one image are sent to the cloud computing platform for the cloud computing platform to update the detection model; the tag of the at least one image is used to indicate that the foreign vehicle is included in the at least one image.

[0037] In one or more embodiments, the communication unit is specifically configured to:

[0038] Establishing a satellite communication link with the data center through satellite network side equipment;

[0039] The vehicle information and location information of the external vehicle are sent to the data center via the satellite communication link.

[0040] In one or more embodiments, the processing unit is further used to train the detection model:

[0041] Acquire a training sample set; the training sample set includes a plurality of labeled sample images;

[0042] Extracting a set number of sample images from the training sample set and combining them to obtain new sample images, and using the new sample images to expand the training sample set;

[0043] The detection model is trained using the expanded training sample set.

[0044] In a third aspect, an electronic device is provided, the electronic device comprising a controller and a memory. The memory is used to store computer-executable instructions, and the controller executes the computer-executable instructions in the memory to use hardware resources in the controller to perform the operation steps of any possible implementation method of the first aspect.

[0045] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is run on the computer, the computer executes the above-mentioned methods.

[0046] The present application proposes that an image acquisition device sends the images of vehicles in a set area to an edge server, which detects and identifies images containing foreign vehicles, and determines the vehicle information and location information of the foreign vehicles based on the identified images, so that the data center can monitor the movement trajectory of foreign vehicles in the set area. Compared with the solution of using a camera device or a cloud computing platform for image detection in traditional technologies, the solution of the present application can effectively improve the real-time performance of image detection, save bandwidth costs, and reduce the cost of the camera device. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application.

[0048] Figure 1 A schematic diagram of the architecture of a YOLO algorithm provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of the implementation process of a Focus model provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the implementation process of a Conv module provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the implementation process of a C3 module provided in an embodiment of the present application;

[0052] Figure 5 A schematic diagram of an implementation process of an SPP module provided in an embodiment of the present application;

[0053] Figure 6 A schematic diagram of an implementation of a mosaic data enhancement method provided in an embodiment of the present application;

[0054] Figure 7 A schematic diagram of the architecture of a foreign vehicle monitoring system provided in an embodiment of the present application;

[0055] Figure 8 A schematic flow chart of a method for monitoring an incoming vehicle provided in an embodiment of the present application;

[0056] Fig. 9A A schematic flow chart of another method for monitoring foreign vehicles provided in an embodiment of the present application;

[0057] Fig. 9B A schematic flow chart of another method for monitoring foreign vehicles provided in an embodiment of the present application;

[0058] Fig.10 A schematic diagram of the structure of a monitoring device for a foreign vehicle provided in an embodiment of the present application;

[0059] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0061] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0062] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0063] In order to facilitate understanding of the solutions provided by the embodiments of the present application, the technical terms involved in the present application are first introduced below:

[0064] (1) YOLO object detection algorithm: The YOLO algorithm uses a separate convolutional neural network (CNN) model to implement object detection. First, the input image is resized to 448*448, and then input into the CNN model to obtain the prediction result, that is, whether the image contains an object. The YOLO algorithm has been developed into multiple versions, including YOLOv1-YOLOv8. This application does not limit the object detection algorithm used. Here, only the YOLOv5 algorithm is introduced as an example.

[0065] (2) YOLOv5 target detection algorithm: The architecture of the YOLOv5 algorithm can be found in Figure 1 , mainly includes three components: Backbone component, Neck component and Head component. Backbone component is mainly used to extract features from images, Neck component is used to mix and combine features and pass features to the prediction layer, and Head component is used to predict and output prediction results.

[0066] like Figure 1 The architecture diagram shown in the figure includes a Focus module which is similar to downsampling and increases the feature multiple by slicing the image. For example, the implementation process of the Focus model can be found in Figure 2The specific operation is to extract a grid every other grid after the image is gridded, and compose an image with the extracted grids, thereby realizing the division of an image into four images. Furthermore, the four images obtained by segmentation are concatenated and convolved (the convolution calculation amount is four times the conventional calculation amount), and four times the features of the original image can be obtained. For example, for the original 640×640×3 image, after slicing, it becomes a 320×320×12 feature map, and after concatenation, it is convolved to obtain a 320×320×64 feature map. Figure 1 The Conv module included in the architecture diagram is used to perform operations such as convolution and activation functions on the image to extract features from the image. For example, the implementation process of the Conv module can be found in Figure 3 ,exist Figure 3 The activation function is shown in the example of Silu function. Of course, the activation function can also use Mish function: f(x) = x*tanh(ln(1+e x )).

[0067] Figure 1 The C3 module included in the shown architecture is used to learn the residual features so that the detection model can learn more features. Figure 4 , is a schematic diagram of the implementation process of a C3 module, which includes two branches, one branch uses a Bottleneck module and three convolution modules, and the other branch uses a convolution module, and finally the two branches are spliced. Among them, the role of the Bottleneck module is to reduce the number of parameters, using multiple small convolution kernels instead of a large convolution kernel, first reducing the number of channels and then expanding them. For example, you can first use 1×1 convolution to reduce the number of channels by half, and then use 3×3 convolution to double the number of channels and obtain features.

[0068] Figure 1 The role of the spatial pyramid pooling (SPP) module included in the architecture shown is to convert feature maps of any size into feature vectors of fixed size. For example, the implementation process of the SPP module can be found in Figure 5 First, the number of input channels is halved through a convolution module, and then they are input into the maximum pooling layer (maxpooling) with kernels of different sizes. The three maximum pooling results are then concatenated with the data without pooling operation, and finally the features obtained from the concatenation interface are calculated by a convolution module.

[0069] (3) Mosaic data enhancement: It is a method used to enrich the training sample set. The specific process is: first, four pictures are randomly scaled, randomly cropped, and randomly arranged to obtain a new image. For example, see Figure 6, which is a schematic diagram of the implementation of a mosaic data enhancement method. Figure 6 In the figure, images A, B, C, and D are combined and spliced, and the part in the dotted box is the new combined image. The mosaic data enhancement method can generate new training samples in this way, enriching the training sample set, and because the spliced ​​images can be randomly scaled and arranged, many small targets are added, which improves the robustness of the detection model.

[0070] (4) CIOU (Complete-IOU): A prediction box accuracy judgment method based on the improved Intersetion Over Union (IOU). IOU calculates (the overlapping area of ​​the predicted box and the actual box) / (the sum of the areas of the predicted box and the actual box), which is used to characterize the accuracy of the predicted box. The higher the IOU value, the higher the accuracy of the predicted box. The IOU loss function refers to taking the logarithm of the obtained IOU value with the base e and adding a negative sign in front of it, which is the IOU loss function. In actual application, researchers have found that the IOU loss function may be the same when the IOU values ​​are different, so the IOU loss function cannot accurately reflect the degree of overlap of the bounding box. Based on this, CIOU is proposed in the related art, which comprehensively considers the overlapping area, center point distance and aspect ratio. For example, the CIOU loss function can be seen in the following formula (1):

[0071]

[0072] Among them, LOSS CIOU represents the loss function of CIOU, IOU represents the intersection-over-union ratio of the predicted box and the actual box, b represents the center point of the predicted box, and b gt represents the center point of the actual box, ρ represents the Euclidean distance between the center points of the predicted box and the actual box, and c represents the length of the diagonal of the minimum closure area that can contain both the predicted box and the actual box. α is a weight function, and the details can be found in the following formula (2). v is used to measure the similarity of the aspect ratio of the predicted box and the actual box. For example, v can be determined by the following formula (3).

[0073]

[0074]

[0075] Among them, w is the length of the prediction box, h is the width of the prediction box, and h gt is the actual frame length, w gt The actual frame width.

[0076] (5) Non-maximum Suppression (NMS): It is an algorithm for removing non-maximum values ​​and is commonly used in edge detection, object recognition and other fields in computer vision. The specific process of NMS is as follows: First, the prediction boxes are sorted in descending order according to their confidence to obtain a candidate box list, and the prediction box A with the highest confidence is selected and added to the output list, and the prediction box A is deleted from the candidate box list. Furthermore, the IOU between the prediction box A and all the prediction boxes in the candidate box list is calculated, and the prediction boxes with IOU greater than the set threshold are deleted. The above process is repeated until the candidate box list is empty and the output list is returned. It can be seen that NMS mainly screens each candidate prediction box by IOU. This method is not friendly to the recognition of two objects that are close to each other in the image, and may ignore any of the two objects that are close to each other. Based on this, Soft-NMS is proposed in the related art. Its idea is that the prediction boxes with an overlap with the prediction box A greater than the threshold will not be deleted, but a function is used to attenuate the confidence of these prediction boxes. The specific process is: calculate the IOU between the prediction box A with the highest confidence and all other prediction boxes, update the confidence of the prediction box whose IOU is less than the set threshold using the preset attenuation function, repeat the above process until the confidence of all remaining prediction boxes is updated, and then filter each prediction box according to the confidence threshold to obtain the output list. It can be seen that compared with NMS, Soft-NMS adds a decay function to update the confidence of the prediction box whose IOU is less than the set threshold, instead of directly deleting it. The decay function can be a linear function or a Gaussian function.

[0077] (6) Label smoothing: A method to prevent overfitting in classification problems. When labeling training samples, two labeling methods are commonly used. One is to use hard labels, which are either 1 or 0. The other is soft labels, where the labeled values ​​are not fixed values. For example, a painting can have a 40% probability of being a cat and a 60% probability of being a dog. In current machine learning, if hard labels are used for labeling, when there are information labeling errors in the data samples, the consequence is repeated mistakes and poor generalization ability. To solve this problem, label smoothing is proposed in traditional technology, which uses soft labels for labeling to reduce overfitting.

[0078] The embodiment of the present application proposes a method for monitoring foreign vehicles, which is used to quickly detect foreign vehicles in a set area and locate and monitor foreign vehicles. In traditional detection schemes, chips deployed on the device side, such as chips in a camera device, are generally used for image detection. This method has high requirements for the camera device and is relatively costly. Another method is that the camera device uploads the collected images to a computing platform in the cloud, and the cloud performs image detection. This method consumes bandwidth resources and has poor real-time performance. Based on this, the present application proposes that image recognition be performed by an edge server deployed at the edge of the network, which can promptly identify foreign vehicles in the set area, locate the position of foreign vehicles, and monitor the movement trajectory of foreign vehicles.

[0079] In order to facilitate understanding of the solution of the present application, the architecture of the foreign vehicle monitoring system applicable to the solution of the present application is introduced below. Figure 7 , is a schematic diagram of the architecture of a monitoring system for foreign vehicles provided in an embodiment of the present application, which includes a cloud computing platform, a data center, an edge server and an image acquisition device.

[0080] Figure 7 The cloud computing platform included in the system shown is used to train and update the image detection model. The functions of the cloud computing platform can be implemented by a server or server cluster deployed in the cloud. The data center is used to monitor the movement trajectory of foreign vehicles in a set area. The functions of the data center can be implemented by a computing device equipped with a display screen. This application does not limit the specific implementation of the computing device, for example, it can be a server, a personal computer, a host, etc.

[0081] Figure 7 The edge server included in the system shown is used to detect images taken in a set area, determine whether the set area includes foreign vehicles, and locate foreign vehicles. The edge server can be understood as sinking the traditional server computing power from the cloud computing platform to the edge server close to the user end, thereby achieving the effect of reducing latency, reducing bandwidth requirements and costs.

[0082] For example, Figure 7 The image acquisition device shown may be an image acquisition device set at a fixed position inside the set area, such as a camera mounted at a key checkpoint. Or it may be an image acquisition device deployed in a vehicle parked or traveling inside the set area, such as a vehicle-mounted camera of the vehicle. For example, Figure 7The image acquisition device shown in can be used to send the acquired image to the edge server for image detection, or can be used to directly perform image detection on the acquired image and send the detection result to the edge server or to the data center. It should be noted that the present application does not limit the number of image acquisition devices included in the monitoring system for foreign vehicles. Figure 7 An image acquisition device is shown only as an example.

[0083] Next, combine Figure 7 The system shown in the figure introduces the solution of this application. Figure 8 , is a flow chart of a method for monitoring a foreign vehicle provided in an embodiment of the present application. For example, Figure 8 The method flow shown can be Figure 7 The system shown in the figure may be executed by the edge server included in the system, or may be executed by a specific component included in the edge server, such as a processor or a processing chip in the edge server. Figure 8 The execution subject of the method flow shown is not limited. Figure 8 The method flow shown specifically includes:

[0084] 801, receiving images sent by multiple image acquisition devices located in a set area.

[0085] Exemplarily, the set area can be understood as an internal area such as a closed park, a control area, etc. The image acquisition device in the set area can send the real-time acquired images to the edge server, or can also send images periodically according to a set period.

[0086] 802 , use a pre-trained detection model to detect multiple images and determine at least one image containing a foreign vehicle.

[0087] The vehicles allowed to enter the set area can be referred to as internal vehicles, and the vehicles outside the internal vehicles are referred to as external vehicles. For example, the detection model can be pre-trained by an edge server or a cloud computing platform, and the trained detection model can extract features from the image to determine whether the image includes an external vehicle.

[0088] 803 , determining vehicle information of the foreign vehicle according to at least one image, and determining position information of the foreign vehicle according to the position of an image acquisition device that acquires at least one image.

[0089] Optionally, the vehicle information may include information such as a license plate number, body color, vehicle model, and vehicle outline of the foreign vehicle. The edge server may extract the vehicle information of each foreign vehicle from at least one image.

[0090] Exemplarily, when receiving multiple images, the edge server may record the location information of the image acquisition device corresponding to each image. For example, when the image acquisition device is an image acquisition device set at a fixed position, the edge server may record the location of each image acquisition device. Alternatively, when the image acquisition device is an image acquisition device set in an internal vehicle, the internal vehicle may carry its own location information when sending images, so that the edge server can determine the location corresponding to each image. Furthermore, after the edge server identifies at least one image containing a foreign vehicle from multiple images, it can determine the acquisition location of at least one image based on the recorded or received location of the image acquisition device, thereby using the acquisition location as the location information of the foreign vehicle contained in the image.

[0091] 804, sending the vehicle information and location information of the external vehicle to the data center.

[0092] The information of the external vehicle is sent to the data center for the data center to monitor the movement trajectory of the external vehicle in the set area.

[0093] Based on the above scheme, the present application proposes that the image acquisition device sends the images of vehicles in the set area to the edge server, and the edge server detects and identifies the images containing foreign vehicles, and determines the vehicle information and location information of the foreign vehicles based on the identified images, so that the data center can monitor the movement trajectory of foreign vehicles in the set area. Compared with the traditional technology of using camera devices or cloud computing platforms for image detection, the solution of the present application can effectively improve the real-time performance of image detection, save bandwidth costs, and reduce the cost of camera devices.

[0094] In some embodiments, before receiving the image sent by the image acquisition device, the edge server can receive the detection model sent by the cloud computing platform. That is, the cloud computing platform can use a pre-labeled training sample set to train the detection model. Among them, the images included in the training sample set can be images obtained from the surveillance video stream in the set area, and each image in the training sample set corresponds to a category label, which is used to characterize whether the image includes foreign vehicles. Optionally, the training sample set can also be split, with one part of the images used to train the detection model and the other part of the images used to verify the accuracy of the trained detection model. Exemplarily, the cloud computing platform can use the YOLOv5 algorithm to train the detection model, so that the structure of the trained detection model can be seen. Figure 1 .

[0095] In one possible implementation, the cloud computing platform can expand the training sample set before training the detection model. For example, the cloud computing platform can extract a set number of sample images from the training sample set and combine them to obtain a new sample image, and add the new sample image to the training sample set to expand the training sample set. As an optional method, the cloud computing platform can use a mosaic data enhancement algorithm to expand the training sample set, where the introduction of the mosaic data enhancement algorithm can be found in the technical terminology introduction section, which will not be repeated here.

[0096] When the YOLOv5 algorithm is used to train the detection model, the images included in the training sample set can first be input into the Focus module in the Backbone component for downsampling, and then feature extraction is performed through multiple convolution modules and C3 modules, and the extracted features are input into the Neck module. After convolution, upsampling and splicing of the features, they are output to the Head module for label prediction and prediction box generation, and the confidence of the prediction box is output. When the Head module makes a prediction, in order to improve the accuracy of the prediction box, the embodiment of the present application proposes that CIOU can be used to determine the confidence of the prediction box. Furthermore, when selecting a prediction box based on confidence, the Soft-NMS algorithm can be used to avoid mistakenly deleting the prediction box where the target exists. Furthermore, when labeling the training samples, the label smoothing method can be used to avoid overfitting.

[0097] After the cloud computing platform sends the trained detection model to the edge server, the edge server can use the detection model to detect the image from the image acquisition device to determine the foreign vehicles in the set area. When the image acquisition device is an image acquisition device deployed in an internal vehicle, in one possible implementation, the image acquisition device can directly send the acquired image to the edge server. For example, the image frame can be extracted from the video acquired by the image acquisition device (such as a vehicle-mounted camera) at a set frequency and sent to the edge server. In another possible implementation, the image acquisition device can also be pre-configured with a trained detection model. For example, the detection model can be pre-configured in the vehicle's on-board computing platform, so that the image acquisition device can use the pre-trained detection model to detect the acquired image, determine the vehicle information and location information of the foreign vehicle, and send the determined vehicle information and location information to the edge server. In another possible implementation method, when the communication link between the image acquisition device and the edge server fails to be established, for example, when the image acquisition device is located in a no-signal area or a signal-blocked area, a communication link cannot be successfully established with the edge server. In this case, the image acquisition device can establish a satellite communication link with the data center through the satellite network side, and then send the vehicle information and location information of the external vehicle to the data center through the satellite communication link.

[0098] In some embodiments, whether the image acquisition device detects images to determine foreign vehicles or the edge server detects images to determine foreign vehicles, the detected images containing foreign vehicles can be sent to the cloud computing platform for the cloud computing platform to expand the training sample set used for training the detection model, so that the cloud computing platform can use the expanded training sample set to update the detection model and send the updated detection model to the image acquisition device and the edge server.

[0099] In some scenarios, the image acquisition device or edge server may be located in a signal transmission restricted area such as an unmanned border area, so the efficiency of using the traditional 4G / 5G network to transmit data to the cloud computing platform and data center is not high. Based on this, the present application proposes that both the image acquisition device and the edge server can establish satellite communication links with the cloud computing platform and data center respectively through the satellite network side. Thus, the image acquisition device and the edge server can send images containing foreign vehicles to the cloud computing platform through the satellite communication link between the cloud computing platform, and the image acquisition device and the edge server can send the detected foreign vehicle's vehicle information and location information to the data center through the satellite communication link established with the data center. Based on the above characteristics, the solution of the present application can be applied to areas such as unmanned areas and borders where the ground network is unstable, and the movement trajectory of foreign vehicles can still be efficiently and stably monitored in these areas.

[0100] In order to further understand the solution proposed in this application, the following is introduced in conjunction with specific embodiments. Fig. 9A , is a flow chart of a method for monitoring a foreign vehicle provided in an embodiment of the present application, specifically comprising:

[0101] 901A, the cloud computing platform uses the training sample set to train the detection model and sends the trained detection model to the edge server.

[0102] Exemplarily, the cloud computing platform can use the YOLOv5 algorithm to train the detection model. The specific process can be found in the introduction of the above embodiment and will not be repeated here.

[0103] 902A, the edge server uses the received detection model to detect the image taken by the image acquisition device set in the set area to determine the vehicle information and location information of the foreign vehicle.

[0104] The image acquisition device may be an image acquisition device disposed at a fixed position within a set area, or may be a vehicle-mounted camera device deployed in an internal vehicle within the set area.

[0105] 903A, the edge server sends the vehicle information and location information of the external vehicle to the data center via the satellite communication link.

[0106] 904A, the data center displays the vehicle information and location information of the external vehicle on the display screen to monitor the movement trajectory of the external vehicle.

[0107] 905A, the edge server sends the detected image containing the foreign vehicle to the cloud computing platform.

[0108] In 906A, the cloud computing platform uses the received image to expand the training sample set, and uses the expanded training sample set to update the detection model.

[0109] Above Fig. 9A The solution of performing image detection by the edge server is shown. For example, the image detection process can also be performed by an internal vehicle located within the set area, such as by an on-board computing platform of the internal vehicle, see Fig. 9B , is a flow chart of another method for monitoring foreign vehicles provided in an embodiment of the present application, which specifically includes:

[0110] 901B, the cloud computing platform uses the training sample set to train the detection model, and sends the trained detection model to the on-board computing platform.

[0111] 902B, the vehicle-mounted computing platform uses the received detection model to detect the image collected by the vehicle-mounted camera device to determine the vehicle information and location information of the foreign vehicle.

[0112] 903B, the vehicle computing platform sends the vehicle information and location information of the external vehicle to the data center via the satellite communication link.

[0113] 904B, the data center displays the vehicle information and location information of the external vehicle on the display screen to monitor the movement trajectory of the external vehicle.

[0114] 905B, the vehicle-mounted computing platform sends the detected image containing the foreign vehicle to the computing platform.

[0115] 906B, the cloud computing platform uses the received image to expand the training sample set, and uses the expanded training sample set to update the detection model.

[0116] Based on the same concept as the above method, see Fig.10 , is a monitoring device 1000 for a foreign vehicle provided in an embodiment of the present application, the device 1000 is used to implement each step in the above method, and in order to avoid repetition, it will not be described again here. The device 1000 includes: a communication unit 1001 and a processing unit 1002.

[0117] The communication unit 1001 is used to receive images sent by multiple image acquisition devices located in a set area;

[0118] The processing unit 1002 is used to detect the multiple images using a pre-trained detection model to determine at least one image containing a foreign vehicle;

[0119] The processing unit 1002 is further configured to determine the vehicle information of the foreign vehicle according to the at least one image, and determine the position information of the foreign vehicle according to the position of the image acquisition device that acquires the at least one image;

[0120] The communication unit 1001 is also used to send the vehicle information and location information of the external vehicle to the data center, so that the data center can monitor the movement trajectory of the external vehicle in the set area.

[0121] In one or more embodiments, the plurality of image acquisition devices include image acquisition devices deployed at fixed positions within the set area, and image acquisition devices deployed in an interior vehicle located within the set area.

[0122] In one or more embodiments, when the plurality of image acquisition devices are image acquisition devices deployed inside a vehicle, the communication unit 1001 is further configured to:

[0123] receiving vehicle information and position information of the external vehicle sent by the multiple image acquisition devices; the vehicle information of the external vehicle sent by the multiple image acquisition devices is obtained by the multiple image acquisition devices detecting the collected images based on the detection model, and the position information of the external vehicle sent by the multiple image acquisition devices is determined by the multiple image acquisition devices according to their own positions;

[0124] The received vehicle information and location information of the external vehicle are forwarded to the data center.

[0125] In one or more embodiments, before receiving images sent by multiple image acquisition devices located in a set area, the communication unit 1001 is further used to:

[0126] Receiving the detection model sent by the cloud computing platform;

[0127] After determining the vehicle information and location information of the foreign vehicle, the communication unit 1001 is further used to:

[0128] The at least one image and a tag of the at least one image are sent to the cloud computing platform for the cloud computing platform to update the detection model; the tag of the at least one image is used to indicate that the foreign vehicle is included in the at least one image.

[0129] In one or more embodiments, the communication unit 1001 is specifically configured to:

[0130] Establishing a satellite communication link with the data center through satellite network side equipment;

[0131] The vehicle information and location information of the external vehicle are sent to the data center via the satellite communication link.

[0132] In one or more embodiments, the processing unit 1002 is further configured to train the detection model:

[0133] Acquire a training sample set; the training sample set includes a plurality of labeled sample images;

[0134] Extracting a set number of sample images from the training sample set and combining them to obtain new sample images, and using the new sample images to expand the training sample set;

[0135] The detection model is trained using the expanded training sample set.

[0136] Fig.11The electronic device 1100 provided in the embodiment of the present application is shown in FIG. 1 . The electronic device 1100 in the embodiment of the present application may further include a communication interface 1103, which is, for example, a network port, through which the electronic device may transmit data. Optionally, the communication interface 1103 may be used to implement the above Fig.10 The function of the communication unit 1001.

[0137] In the embodiment of the present application, the memory 1102 stores instructions that can be executed by at least one controller 1101. The at least one controller 1101 can be used to execute each step in the above method by executing the instructions stored in the memory 1102. For example, the controller 1101 can implement the above Fig.10 The function of the processing unit 1002 in.

[0138] Among them, the controller 1101 is the control center of the electronic device, which can use various interfaces and lines to connect various parts of the entire electronic device, by running or executing instructions stored in the memory 1102 and calling data stored in the memory 1102. Optionally, the controller 1101 may include one or more processing units, and the controller 1101 may integrate an application controller and a modem controller, wherein the application controller mainly processes the operating system and application programs, etc., and the modem controller mainly processes wireless communications. It is understandable that the above-mentioned modem controller may not be integrated into the controller 1101. In some embodiments, the controller 1101 and the memory 1102 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0139] The controller 1101 may be a general controller, such as a central processing unit (CPU), a digital signal controller, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, which may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general controller may be a microcontroller or any conventional controller, etc. The steps performed by the data statistics platform disclosed in the embodiments of the present application may be performed directly by a hardware controller, or by a combination of hardware and software modules in the controller.

[0140] The memory 1102 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1102 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (English: Random Access Memory, referred to as: RAM), a static random access memory (English: Static Random Access Memory, referred to as: SRAM), a programmable read-only memory (English: Programmable Read Only Memory, referred to as: PROM), a read-only memory (English: Read Only Memory, referred to as: ROM), an electrically erasable programmable read-only memory (English: Electrically Erasable Programmable Read-Only Memory, referred to as: EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 1102 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1102 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0141] By designing and programming the controller 1101, for example, the code corresponding to the method introduced in the above embodiment can be solidified into the chip, so that the chip can execute the steps of the above method during operation. How to design and program the controller 1101 is a technology well known to those skilled in the art and will not be repeated here.

[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0143] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a controller of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the controller of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0146] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0147] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for monitoring an incoming vehicle, characterized in that: The method is applied to an edge server and includes: receiving images sent by a plurality of image acquisition devices located in a set area; Using a pre-trained detection model to detect multiple images, and determining at least one image containing a foreign vehicle; determining vehicle information of the foreign vehicle based on the at least one image, and determining position information of the foreign vehicle based on the position of an image acquisition device that acquires the at least one image; The vehicle information and location information of the external vehicle are sent to the data center, so that the data center can monitor the movement track of the external vehicle in the set area.

2. The method according to claim 1, characterized in that The plurality of image acquisition devices include image acquisition devices deployed at fixed positions within the set area, and image acquisition devices deployed in an interior vehicle located within the set area.

3. The method according to claim 2, characterized in that When the plurality of image acquisition devices are image acquisition devices deployed inside a vehicle, the method further includes: receiving vehicle information and position information of the external vehicle sent by the multiple image acquisition devices; the vehicle information of the external vehicle sent by the multiple image acquisition devices is obtained by the multiple image acquisition devices detecting the collected images based on the detection model, and the position information of the external vehicle sent by the multiple image acquisition devices is determined by the multiple image acquisition devices according to their own positions; The received vehicle information and location information of the external vehicle are forwarded to the data center.

4. The method according to any one of claims 1 to 3, characterized in that: Before receiving images sent by a plurality of image acquisition devices located in a set area, the method further includes: Receiving the detection model sent by the cloud computing platform; After determining the vehicle information and location information of the foreign vehicle, the method further includes: The at least one image and a tag of the at least one image are sent to the cloud computing platform for the cloud computing platform to update the detection model; the tag of the at least one image is used to indicate that the foreign vehicle is included in the at least one image.

5. The method according to any one of claims 1 to 3, characterized in that: The sending the vehicle information and location information of the external vehicle to the data center server includes: Establishing a satellite communication link with the data center through satellite network side equipment; The vehicle information and location information of the external vehicle are sent to the data center via the satellite communication link.

6. The method according to any one of claims 1 to 3, characterized in that: The detection model is trained in the following manner: Acquire a training sample set; the training sample set includes a plurality of labeled sample images; Extracting a set number of sample images from the training sample set and combining them to obtain new sample images, and using the new sample images to expand the training sample set; The detection model is trained using the expanded training sample set.

7. A monitoring device for foreign vehicles, characterized in that: The device is an edge server, or the device is applied to the edge server, and the device includes: A communication unit, used for receiving images sent by a plurality of image acquisition devices located in a set area; A processing unit, configured to detect the plurality of images using a pre-trained detection model, and determine at least one image containing a foreign vehicle; The processing unit is further configured to determine the vehicle information of the foreign vehicle based on the at least one image, and determine the position information of the foreign vehicle based on the position of an image acquisition device that acquires the at least one image; The communication unit is also used to send the vehicle information and location information of the external vehicle to the data center, so that the data center can monitor the movement trajectory of the external vehicle in the set area.

8. The device according to claim 7, characterized in that The plurality of image acquisition devices include image acquisition devices deployed at fixed positions within the set area, and image acquisition devices deployed in an interior vehicle located within the set area.

9. The device according to claim 8, characterized in that When the plurality of image acquisition devices are image acquisition devices deployed in an internal vehicle, the communication unit is further configured to: receiving vehicle information and position information of the external vehicle sent by the multiple image acquisition devices; the vehicle information of the external vehicle sent by the multiple image acquisition devices is obtained by the multiple image acquisition devices detecting the collected images based on the detection model, and the position information of the external vehicle sent by the multiple image acquisition devices is determined by the multiple image acquisition devices according to their own positions; The received vehicle information and location information of the external vehicle are forwarded to the data center.

10. The device according to any one of claims 7 to 9, characterized in that: Before receiving images sent by a plurality of image acquisition devices located in a set area, the communication unit is further used to: Receiving the detection model sent by the cloud computing platform; After determining the vehicle information and location information of the foreign vehicle, the communication unit is further used to: The at least one image and a tag of the at least one image are sent to the cloud computing platform for the cloud computing platform to update the detection model; the tag of the at least one image is used to indicate that the foreign vehicle is included in the at least one image.

11. The device according to any one of claims 7 to 9, characterized in that: The communication unit is specifically used for: Establishing a satellite communication link with the data center through satellite network side equipment; The vehicle information and location information of the external vehicle are sent to the data center via the satellite communication link.

12. The device according to any one of claims 7 to 9, characterized in that: The processing unit is further used to train the detection model: Acquire a training sample set; the training sample set includes a plurality of labeled sample images; Extracting a set number of sample images from the training sample set and combining them to obtain new sample images, and using the new sample images to expand the training sample set; The detection model is trained using the expanded training sample set.

13. An electronic device, characterized in that: include: Memory and controller; A memory for storing program instructions; A controller is used to call the program instructions stored in the memory and execute the method according to any one of claims 1 to 6 according to the obtained program.

14. A computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are used to execute the method according to any one of claims 1-6.

Images