Method, system and device for supervising pledged vehicles and computer equipment

Abstract

The embodiment of the invention provides a pledged vehicle supervision method, system and device and computer equipment. The method comprises the steps that image data, collected by an unmanned aerial vehicle camera, of a pledged vehicle is acquired; inputting the image data into a vehicle detection model to obtain quantity information of pledged vehicles; obtaining the frame number information of the pledged vehicle collected by an Internet of Things gateway device, wherein the Internet of Things gateway device is in communication connection with an Internet of Things device on the pledged vehicle; comparing the quantity information and the frame number information with pre-stored pledged vehicle information; and generating alarm information according to a comparison result and feeding back the alarm information to a supervisor. According to the method, the quantity information obtained by the unmanned aerial vehicle camera and the vehicle frame number information collected by the Internet of Things gateway equipment are combined to form multi-modal data information to perform fusion supervision on the pledge vehicles in the warehouse, so that the supervision strength and accuracy are improved, the workload of supervisors is reduced, the supervision efficiency of the pledge vehicles in the warehouse is improved, and the supervision efficiency of the pledge vehicles in the warehouse is improved. And asset safety can be guaranteed.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of intelligent supervision technology, and in particular to a pledged vehicle supervision method, system, device and computer equipment. Background Art

[0002] With the continuous development of computer technology, more and more traditional supervision methods are now using digital means. For example, in the scenario of automobile pledge supervision, an automobile pledge supervision system has been built. Through the system accounting of business operations such as vehicle entry, daily inventory, vehicle release, and vehicle transfer, digital supervision of pledged vehicles can be achieved. The supervision system can adopt a multi-system docking method, such as docking with the bank system and the logistics company system, so as to achieve rapid transmission of online system information and reduce manual export and import work. Offline supervision uses the method of deploying cameras to monitor the site. When an abnormal situation occurs, the surveillance video is manually retrieved for investigation. In addition, RFID tags can be used to check the vehicle inventory, which also plays a role in improving the efficiency of inventory to a certain extent.

[0003] However, the above-mentioned supervision method for pledged vehicles still has certain defects: for example, when counting vehicles, supervisors are still required to manually use handheld devices to conduct offline counting of vehicles. When the number of vehicles is large, the workload of supervisors will be increased and the counting cycle will also increase accordingly; when there is a problem of insufficient supervision, it is difficult to prevent irregular behaviors. For example, if the pledged vehicle leaves the supervision warehouse after this manual counting and before the next counting begins, and the abnormal situation of the vehicle is not discovered in time, it will lead to the loss of assets.

[0004] In view of this, the embodiments of this specification are intended to provide a pledged vehicle supervision method, system, device and computer equipment. Summary of the invention

[0005] In view of the above problems in the prior art, the purpose of the embodiments of this specification is to provide a pledged vehicle supervision method, system, device and computer equipment to solve the problem of low efficiency and strength of pledged vehicle supervision in the prior art.

[0006] In order to solve the above technical problems, the specific technical solutions of the embodiments of this specification are as follows:

[0007] In a first aspect, an embodiment of the present specification provides a pledge vehicle supervision method, including:

[0008] Obtain image data of pledged vehicles stored in the warehouse collected by drone cameras;

[0009] Input the image data into a pre-trained vehicle detection model to obtain the quantity information of the pledged vehicles output by the vehicle detection model;

[0010] Acquire the vehicle frame number information of the pledged vehicle collected by an IoT gateway device, wherein the IoT gateway device is integrated on the drone and is in communication connection with an IoT device pre-installed on the pledged vehicle;

[0011] Comparing the quantity information and the frame number information of the pledged vehicles with the pre-stored pledged vehicle information respectively;

[0012] Based on the comparison results, an alarm message is generated and fed back to the supervisor.

[0013] Specifically, before inputting the image data into a pre-trained vehicle detection model, the method further includes:

[0014] The image data is spliced.

[0015] Furthermore, the image data is spliced, including:

[0016] Sorting the image frames according to the acquisition time of each image frame in the image data;

[0017] Performing feature point detection on each of the image frames using a feature point algorithm to obtain feature points of each of the image frames;

[0018] Based on the feature points, feature point matching is performed on any two image frames whose acquisition time is adjacent to each other to obtain a pair of successfully matched feature points;

[0019] According to the feature point pairs, any two image frames that are adjacent in acquisition time are spliced ​​to obtain spliced ​​image data.

[0020] Furthermore, based on the feature points, feature point matching is performed on any two image frames that are acquired in adjacent time periods to obtain a successfully matched feature point pair, including:

[0021] Calculating the distance between any feature point in one of any two image frames adjacent in acquisition time and each feature point in the other of any two image frames adjacent in acquisition time;

[0022] Determine a first feature point in another image frame that is closest to any one of the feature points;

[0023] Determine whether the distance between the first feature point and any one of the feature points is less than a preset distance threshold;

[0024] If yes, determine a second feature point in another image frame that is next closest to the arbitrary feature point, and calculate a distance coefficient between the first feature point and the second feature point;

[0025] Determining whether the distance coefficient is less than a preset distance coefficient threshold;

[0026] If yes, determining that the first feature point and any one of the feature points form a feature point pair;

[0027] Otherwise, it is determined that no feature point matching any one of the feature points exists in the other image frame.

[0028] Specifically, according to the feature point pair, any two image frames that are adjacent in acquisition time are spliced ​​to obtain spliced ​​image data, including:

[0029] Determining, based on the feature point pairs, a homography matrix between any two image frames that are adjacent in acquisition time;

[0030] According to the homography matrix, any two image frames that are adjacent in acquisition time are spliced ​​to obtain spliced ​​image data.

[0031] Specifically, the vehicle detection model is trained through the following steps:

[0032] Acquire vehicle image training data, and label vehicles in the vehicle image training data;

[0033] Inputting the vehicle image training data into an initial vehicle detection model to obtain a recognition result of the initial vehicle detection model on the vehicle in the vehicle image training data;

[0034] The initial vehicle detection model is iteratively trained according to the labeling results and the recognition results until the recognition rate of the initial vehicle detection model for the vehicle reaches a preset recognition rate threshold or reaches a preset number of iterative training times, thereby obtaining a trained vehicle detection model.

[0035] Preferably, the method further comprises:

[0036] Acquire the target vehicle and pre-defined electronic fence;

[0037] Determine the position of the image frame where the target vehicle is located in the image data according to the vehicle frame number information;

[0038] Determining a storage location of the target vehicle in the warehouse according to a location of the image frame where the target vehicle is located in the image data;

[0039] According to the storage location of the vehicle in the warehouse, determining whether the storage location is located within the electronic fence;

[0040] If not, an alarm message is generated and fed back to the supervisor.

[0041] Furthermore, based on the comparison results, alarm information is generated and fed back to the supervisor, including:

[0042] Determine whether the comparison result between the quantity information and the number of vehicles in the pledged vehicle information is consistent;

[0043] If they are consistent, determining whether the comparison result of the frame number information and the frame numbers of each vehicle in the pledged vehicle information is consistent;

[0044] If yes, it is determined that there is no abnormality in the pledged vehicle;

[0045] Otherwise, the alarm information is generated and fed back to the supervisor.

[0046] In a second aspect, the embodiment of this specification provides a one-stop pledge vehicle supervision system, including:

[0047] A drone, wherein the drone is equipped with a camera for collecting image data of pledged vehicles stored in the warehouse;

[0048] An IoT gateway device, which is integrated on the drone and is used to communicate with an IoT device pre-installed on the pledged vehicle to collect the frame number information of the pledged vehicle;

[0049] A supervision system is connected to the drone and the IoT gateway device, and is used to supervise the pledged vehicles stored in the warehouse based on the image data fed back by the drone, the frame number information fed back by the IoT gateway device and a pledged vehicle supervision method provided by the above technical solution.

[0050] In a third aspect, the embodiments of this specification provide a pledge vehicle supervision device, including:

[0051] The first acquisition module is used to acquire image data of pledged vehicles stored in the warehouse collected by the drone camera;

[0052] An input module, used to input the image data into a pre-trained vehicle detection model to obtain the quantity information of the pledged vehicles output by the vehicle detection model;

[0053] A second acquisition module is used to acquire the frame number information of the pledged vehicle collected by an IoT gateway device, wherein the IoT gateway device is integrated on the drone and is in communication connection with an IoT device pre-installed on the pledged vehicle;

[0054] A comparison module, used to compare the quantity information of the pledged vehicles and the frame number information with the pre-stored pledged vehicle information respectively;

[0055] The alarm feedback module is used to generate alarm information based on the comparison results and feed it back to the supervisor.

[0056] In a fourth aspect, an embodiment of the present specification provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided by the above technical solution when executing the computer program.

[0057] In a fifth aspect, an embodiment of the present specification provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method provided by the above technical solution is implemented.

[0058] In a sixth aspect, an embodiment of the present specification provides a computer program product, comprising at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the method provided by the above technical solution.

[0059] By adopting the above technical solution, the pledged vehicle supervision method, system, device and computer equipment provided in the embodiments of this specification use drone cameras to collect image data of pledged vehicles in the warehouse, and use the Internet of Things gateway device integrated on the drone to collect the frame number information of each pledged vehicle in the warehouse, so that the supervisor does not need to use handheld devices to manually check the pledged vehicles, which reduces the workload of the supervisor and improves the efficiency of supervision of pledged vehicles in the warehouse. And by combining quantity information and frame number information, multimodal data information is formed to integrate and supervise the pledged vehicles in the warehouse, which improves the intensity and accuracy of supervision, plays a role in preventing dishonest behavior, and is conducive to ensuring asset security.

[0060] In order to make the above and other purposes, features and advantages of the embodiments of this specification more obvious and easy to understand, the following specifically cites preferred embodiments and describes them in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 A schematic diagram showing the steps of a pledge vehicle supervision method provided in an embodiment of this specification is shown;

[0063] Figure 2 A schematic diagram showing steps of splicing image data in an embodiment of this specification is shown;

[0064] Figure 3A schematic diagram of the steps of performing feature point matching on any two image frames with adjacent acquisition times based on feature points to obtain a successfully matched feature point pair in an embodiment of this specification is shown;

[0065] Figure 4 A schematic diagram showing the steps of training a vehicle detection model in an embodiment of this specification is shown;

[0066] Figure 5 A schematic diagram showing the steps of another pledged vehicle supervision method provided in an embodiment of this specification is shown;

[0067] Figure 6 A schematic diagram of the steps of generating alarm information according to the comparison result and feeding back to the supervisor in the embodiment of this specification is shown;

[0068] Figure 7 A schematic diagram of the structure of a pledged vehicle supervision device provided in an embodiment of this specification is shown;

[0069] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of this specification is shown.

[0070] Description of the accompanying symbols:

[0071] 71. A first acquisition module;

[0072] 72. Input module;

[0073] 73. Second acquisition module;

[0074] 74. Comparison module;

[0075] 75. Alarm feedback module;

[0076] 802. Computer equipment;

[0077] 804, processor;

[0078] 806. Memory;

[0079] 808, driving mechanism;

[0080] 810, input / output module;

[0081] 812. Input device;

[0082] 814. Output device;

[0083] 816. Presentation equipment;

[0084] 818. Graphical user interface;

[0085] 820, network interface;

[0086] 822, communication link;

[0087] 824. Communication bus. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0089] It should be noted that the terms "first", "second", etc. in this specification and claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0090] In order to solve the problems existing in the existing pledged vehicle supervision methods, such as delayed supervision information, heavy workload for supervisors, low supervision efficiency and insufficient supervision, the embodiments of this specification provide a pledged vehicle supervision method, system, device and computer equipment. Figure 1 It is a schematic diagram of the steps of a pledged vehicle supervision method provided in the embodiments of this specification. This specification provides the method operation steps as described in the embodiments or flow charts, but more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the steps among many orders, and does not represent the only execution order. When the system or device product is executed in practice, it can be executed in the order of the methods shown in the embodiments or the accompanying drawings or in parallel. It should be noted that the pledged vehicle supervision method, system, device and computer equipment provided in the embodiments of this specification can be used in the financial field, and can also be used in any field other than the financial field. The embodiments of this specification do not limit the application field of the provided pledged vehicle supervision method, system, device and computer equipment.

[0091] A pledge vehicle supervision method provided in the embodiments of this specification can be executed by a supervision system. Specifically, Figure 1 As shown, the method may include:

[0092] S110: Obtain image data of pledged vehicles stored in the warehouse collected by the drone camera.

[0093] S120: Input the image data into a pre-trained vehicle detection model to obtain the quantity information of the pledged vehicles output by the vehicle detection model.

[0094] The data collected by drones can be uploaded to the supervision system to provide data services for the supervision system, which is conducive to reducing the integration difficulty and transformation cost of the existing supervision system. At the same time, the number of pledged vehicles can be output through the vehicle detection model, which is conducive to improving the efficiency and accuracy of pledged vehicle counting.

[0095] S130: Acquire the vehicle frame number information of the pledged vehicle collected by an Internet of Things gateway device, wherein the Internet of Things gateway device is integrated on the drone and is communicatively connected with an Internet of Things device pre-installed on the pledged vehicle.

[0096] That is, when a vehicle is pledged and put into storage, each vehicle is bound to an IoT device through physical operation, and the unique identification code of the IoT device is bound to the frame number of the corresponding vehicle in the storage in the supervision system, so that the IoT device and the pledged vehicle form a one-to-one correspondence. Therefore, the information of each IoT device connected to the IoT device and the information of the pledged vehicle corresponding to each IoT device can be viewed in the supervision system.

[0097] Since the IoT gateway device is integrated on the drone, when the drone flies over the warehouse to collect image data, the IoT gateway device can communicate with the IoT devices pre-installed on each pledged vehicle stored in the warehouse to obtain the frame number information of each pledged vehicle. That is, the pledged vehicle supervision method provided in the embodiment of this specification can simultaneously realize the collection of image data and frame number information, greatly improving the collection efficiency, thereby improving the supervision efficiency of the pledged vehicles.

[0098] Specifically, the IoT gateway device and the IoT devices pre-installed on the pledged vehicle can communicate based on the Bluetooth protocol.

[0099] In other feasible embodiments, at least one IoT gateway device may be installed in the warehouse through facilities such as poles, and the number and location distribution of the IoT gateway devices in the warehouse may be determined according to the signal coverage of each IoT gateway device. Therefore, in some supervision scenarios of pledged vehicles with flight restrictions, it is also possible to implement inventory supervision of pledged vehicles only through at least one IoT gateway device installed in the warehouse.

[0100] S140: Compare the quantity information and the frame number information of the pledged vehicles with the pre-stored pledged vehicle information respectively.

[0101] In the embodiments of this specification, the pre-stored pledged vehicle information is the information entered into the supervision system when the vehicle is pledged and put into storage, including the vehicle storage time, vehicle factory date, engine number, certificate number, vehicle color, pledge price, etc. It may also include a picture taken when the vehicle is put into storage, and the unique identification code of the IoT device bound to the vehicle. When the pledged vehicle is out of storage, its pledged vehicle information is deleted from the supervision system.

[0102] S150: Generate an alarm message based on the comparison result and feed it back to the supervisor.

[0103] The embodiment of this specification provides a pledge vehicle supervision method, which uses a drone camera to collect image data of pledged vehicles in a warehouse, and uses an IoT gateway device integrated on the drone to collect the frame number information of each pledged vehicle in the warehouse. There is no need for supervisors to manually check the pledged vehicles with handheld devices, which greatly reduces the workload of supervisors and greatly improves the supervision efficiency of pledged vehicles in the warehouse. And the drone camera is used to collect image data to obtain quantity information, which realizes the "visual" supervision of pledged vehicles; the IoT gateway device is used to collect frame number information, which realizes the "tactile" supervision of pledged vehicles. Combining quantity information and frame number information, multimodal data information is formed to integrate and supervise the pledged vehicles in the warehouse, which can avoid the situation that the number of pledged vehicles is equal, but the actual detained vehicles are not pledged vehicles, and the occurrence of "stealing and replacing pillars"; and it can avoid the problem that the actual detained vehicles are not counted due to the IoT devices installed on the pledged vehicles not being in normal working state, which improves the intensity and accuracy of supervision, plays a role in preventing dishonest behavior, and is conducive to ensuring asset security.

[0104] It should be noted that the pledged vehicle supervision method provided in the embodiments of this specification can be connected to multiple warehouses by the supervision system. Each warehouse needs to be equipped with a drone and an IoT gateway device integrated on the drone; the pledged vehicles stored in warehouses in various places need to be pre-installed with IoT devices that communicate with the IoT gateway devices. The supervision system can supervise the pledged vehicles in each warehouse based on the data information and frame number information of the pledged vehicles in the warehouse. Supervisors can also view the information and working status of the IoT gateway devices corresponding to each warehouse on the supervision system; the IoT gateway devices can be deleted, activated, frozen, unfrozen, and other operations can be performed. The information of the IoT gateway device includes the warehouse it is responsible for supervising (warehouse name, warehouse address), the ID information of the IoT gateway device, and the information of each IoT device that is communicated with the IoT gateway device (including the unique identification code of the IoT device, the vehicle information such as the frame number of the corresponding pledged vehicle, and the warehouse where it is located).

[0105] Furthermore, in the embodiment of this specification, before inputting the image data into a pre-trained vehicle detection model, the method further includes:

[0106] The image data is spliced.

[0107] The overhead images of the supervision area in the warehouse are stitched and integrated through the image stitching algorithm to reduce the repeated areas, and the stitched image data is then input into the vehicle detection model to improve the accuracy of counting the pledged vehicles in the warehouse.

[0108] like Figure 2 As shown, in the embodiment of this specification, the image data is stitched together and may include the following steps:

[0109] S210: Sort the image frames in the image data according to the acquisition time of the image frames.

[0110] That is, the image data may be a collection of image frames captured by the camera of the drone at predetermined intervals when the drone flies along a predetermined trajectory above the warehouse.

[0111] S210: Detect feature points of each image frame using a feature point algorithm to obtain feature points of each image frame.

[0112] In the embodiments of this specification, the feature point algorithm may include but is not limited to features from accelerated segmentation test (FAST) algorithm, scale-invariant feature transform (SIFT) algorithm, accelerated robust features (SURF) algorithm, Harris corner detection algorithm, ORB (Oriented Fast and Rotated Brief) algorithm, Hessian-Laplace algorithm, directed ring feature point algorithm, etc.

[0113] The feature point algorithm used in the embodiments of this specification is not specifically limited. Using the feature point algorithm, feature point detection is performed on each image frame in turn to obtain feature points corresponding to each image frame. Generally, each image frame usually contains multiple feature points.

[0114] In the embodiments of this specification, it is necessary to extract feature points from two image frames that are acquired adjacently at the same time. To increase the success rate of feature point recognition, there needs to be a certain proportion of overlapping area between any two image frames that are acquired adjacently at the same time. Therefore, it is necessary to set the flight speed of the drone over the warehouse and the frequency of the camera taking pictures of the warehouse.

[0115] S230: performing feature point matching on any two image frames with adjacent acquisition times based on the feature points to obtain a successfully matched feature point pair.

[0116] Exemplarily, there are two image frames with adjacent acquisition times, namely image frame A and image frame B. Feature points A1, A2, ..., An are detected by a feature point detection algorithm in image frame A; feature points B1, B2, ..., Bn are detected by a feature point detection algorithm in image frame B. If feature point A1 in image frame A matches feature point B1 in image frame B successfully, a feature point pair (A1, B1) is formed.

[0117] S240: splicing any two image frames that are adjacent in acquisition time according to the feature point pairs to obtain spliced ​​image data.

[0118] It should be noted that the number of successfully matched feature point pairs in any two image frames acquired at adjacent times may not be unique. If the number of feature point pairs is not unique, any two image frames acquired at adjacent times are spliced ​​based on all successfully matched feature point pairs.

[0119] In the embodiments of the present specification, feature points in each image frame are detected by a feature point algorithm and then feature point matching is performed, which can better compensate for the problem of deviation in the collected image frames due to drone flight jitter or inconsistent flight altitude, and is beneficial to improving the stitching effect of the image frames, thereby improving the accuracy of the output of the pledged vehicle quantity information for the stitched image data.

[0120] In a possible embodiment, Figure 3 As shown, step S230: performing feature point matching on any two image frames with adjacent acquisition times based on the feature points to obtain a successfully matched feature point pair may further include the following steps:

[0121] S310: Calculate the distance between any feature point in one of any two image frames adjacent to each other in acquisition time and each feature point in the other of any two image frames adjacent to each other in acquisition time.

[0122] For example, for image frame A and image frame B that are acquired at adjacent times, image frame A includes feature points A1, A2, ..., An; and image frame B includes feature points B1, B2, ..., Bn. For any feature point Ai in image frame A, the distance between it and each feature point in image frame B is calculated respectively.

[0123] S320: Determine a first feature point in another image frame that is closest to any one of the feature points.

[0124] That is, a feature point in the image frame B that is closest to any feature point Ai is recorded as the first feature point.

[0125] S330: Determine whether the distance between the first feature point and any one of the feature points is less than a preset distance threshold.

[0126] Assume that the first feature point is Bj, and record the distance between the first feature point Bj and any feature point Ai in the image frame A as Lj, and determine whether the distance Lj is less than a preset distance threshold.

[0127] S340: If yes, determine a second feature point in another image frame that is next closest to the any one feature point, and calculate a distance coefficient between the first feature point and the second feature point.

[0128] If the distance Lj is less than the preset distance threshold, it indicates that the similarity between the first feature point Bj in the image frame and any feature point Ai in the image frame A is high; it is necessary to obtain the feature point in the image frame B that is the second closest to the any feature point Ai. Then, the feature point in the image frame B that is the second closest to the any feature point Ai is recorded as the second feature point.

[0129] S350: Determine whether the distance coefficient is less than a preset distance coefficient threshold.

[0130] Assuming that the second feature point is Bk, and the distance between the second feature point Bk and any feature point Ai in the image frame A is Lk, the ratio of the distance Lj to the distance Lk is calculated as the distance coefficient, and it is determined whether the distance coefficient is less than the preset distance coefficient threshold.

[0131] S360: If yes, determine that the first feature point and any one of the feature points form a feature point pair.

[0132] If so, it indicates that the similarity between the first feature point and the arbitrary feature point Ai in the image frame A is much greater than the similarity between the second feature point and the arbitrary feature point Ai, thereby determining that the first feature point and the arbitrary feature point constitute a feature point pair.

[0133] S370: Otherwise, determining that there is no feature point matching any one of the feature points in the other image frame.

[0134] In the embodiments of the present specification, the successfully matched feature point pairs in two image frames with adjacent acquisition times are determined through distance calculation and distance coefficient calculation, which is beneficial to improving the accuracy of feature point pair matching, and further beneficial to the accuracy of subsequent stitching of the two image frames, avoiding overlapping parts in the final image data.

[0135] In other feasible embodiments, the image data may also be a video of the warehouse taken by a drone while flying over the warehouse along a predetermined trajectory. Since the video contains a large amount of image frame data, it is also necessary to deduplicate the image frames in the video, and then perform feature point detection and feature point matching and image frame splicing on the deduplicated image frames to improve the efficiency of image frame splicing.

[0136] Furthermore, in the embodiment of this specification, step S240: splicing any two image frames that are adjacent in acquisition time according to the feature point pair to obtain spliced ​​image data may further include:

[0137] Determining, based on the feature point pairs, a homography matrix between any two image frames that are adjacent in acquisition time;

[0138] The homography matrix describes the transformation relationship from one image frame to another image frame in two image frames that are acquired adjacently in time. In the embodiments of this specification, the homography matrix can be calculated using existing calculation methods, which will not be described in detail here.

[0139] According to the homography matrix, any two image frames that are adjacent in acquisition time are spliced ​​to obtain spliced ​​image data.

[0140] It should be noted that in a pledged vehicle supervision method provided in an embodiment of this specification, when splicing two image frames, an image frame with a later acquisition time can be spliced ​​onto an image frame with an earlier acquisition time, thereby completing the splicing of all image frames in the same splicing order.

[0141] In some other feasible embodiments, step S240: splicing any two image frames that are adjacent in acquisition time according to the feature point pair to obtain spliced ​​image data may be:

[0142] Determining the amount of stitching between any two image frames whose acquisition times are adjacent according to the distance between two feature points included in each feature point pair;

[0143] According to the stitching amount, any two image frames that are adjacent in acquisition time are stitched to obtain stitched image data.

[0144] Exemplarily, any two image frames A and image frames B that are adjacent in acquisition time include multiple feature point pairs (A1, B1), (A2, B2)...(An, Bn), then the stitching amount between image frames A and image frames B is determined based on the distance between feature point A1 and feature point B1, the distance between feature point A2 and feature point B2...the distance between feature point An and feature point Bn, and then stitching is performed.

[0145] The stitching amount between two adjacent image frames is determined according to the distance between two feature points in each feature point pair, which can comprehensively consider the influence of each feature point pair on the image frame stitching, which is beneficial to reducing the image frame stitching error and improving the accuracy of the stitched image data.

[0146] Furthermore, if Figure 4 As shown, in the embodiment of this specification, the vehicle detection model is trained by the following steps:

[0147] S410: Acquire vehicle image training data, and label vehicles in the vehicle image training data.

[0148] Exemplarily, each vehicle in the vehicle image training data may be labeled by manual labeling.

[0149] S420: Input the vehicle image training data into an initial vehicle detection model to obtain a recognition result of the initial vehicle detection model on the vehicle in the vehicle image training data.

[0150] In the embodiments of this specification, the initial vehicle detection model may adopt the YOLO v5 detection model, and the vehicle image training data may adopt the open source UA-DETRAC dataset.

[0151] S430: Iteratively train the initial vehicle detection model according to the labeling results and the recognition results until the recognition rate of the initial vehicle detection model for the vehicle reaches a preset recognition rate threshold or reaches a preset number of iterative training times, thereby obtaining a trained vehicle detection model.

[0152] The initial vehicle detection model is trained and optimized in a pre-training manner, which can quickly converge the vehicle detection model and apply it to the pledged vehicle supervision scenario of this method more quickly, thereby achieving accurate and rapid detection of pledged vehicles and reducing missed detections and false detections.

[0153] In some other feasible embodiments, the method may further include:

[0154] S510: Acquire a target vehicle and a pre-defined electronic fence.

[0155] In some specific application scenarios, there may be some pledged vehicles that require key supervision. In this case, they are identified as target vehicles and the electronic fence of the target vehicle is demarcated in advance, that is, the target vehicle is stored in the corresponding electronic fence.

[0156] S520: Determine the position of the image frame where the target vehicle is located in the image data according to the vehicle frame number information.

[0157] When the IoT gateway device collects the frame number information of the target vehicle, the frame number information collection time can be fed back to the supervision system, so that the supervision system can determine the corresponding image frame according to the collection time of the frame number information, thereby determining the position of the image frame where the frame number information is located in the final spliced ​​image data.

[0158] S530: Determine the storage location of the target vehicle in the warehouse according to the location of the image frame where the target vehicle is located in the image data.

[0159] Since the final spliced ​​image data corresponds to the warehouse, the storage location of the vehicle in the warehouse can be determined based on the position of the image frame where the target vehicle is located in the image data.

[0160] S540: According to the storage location of the vehicle in the warehouse, determine whether the storage location is located in the electronic fence.

[0161] S550: If not, an alarm message is generated and fed back to the supervisor.

[0162] If the target vehicle is not stored in the electronic fence, it indicates that there is a certain possibility that the target vehicle will be driven out, so an alarm message is generated and fed back to the supervisor to manually supervise the target vehicle.

[0163] Furthermore, the method further comprises:

[0164] If the frame number information of the target vehicle is not collected, it means that the target vehicle is not stored in the warehouse or the IoT device pre-installed on the target vehicle is not in normal working condition. Therefore, an alarm message needs to be generated and fed back to the supervisor.

[0165] Furthermore, if Figure 6 As shown, in the embodiment of this specification, step S150: generating an alarm message based on the comparison result and feeding it back to the supervisor, further includes:

[0166] S610: Determine whether the comparison result between the quantity information and the number of vehicles in the pledged vehicle information is consistent.

[0167] S620: If they are consistent, determine whether the comparison result of the frame number information and the frame numbers of each vehicle in the pledged vehicle information is consistent.

[0168] In the embodiments of this specification, when the quantity information determined by the image data collected by the drone camera is consistent with the vehicle data of the pledged vehicle information in the supervision system, it is not simply considered that there is no abnormality in the vehicle to avoid possible malicious replacement behavior, that is, replacing a higher-value vehicle with a lower-value vehicle for pledge. The frame number information is further compared, that is, whether the frame number information of the vehicle collected by the IoT gateway corresponds to the frame number information of the pledged vehicle pre-stored in the supervision system.

[0169] S630: If yes, it is determined that there is no abnormality with the pledged vehicle.

[0170] That is, only when the number of pledged vehicles is consistent and the frame number information of the pledged vehicles is also consistent, can it be determined that there is no abnormality in the pledged vehicles.

[0171] S640: Otherwise, the alarm information is generated and fed back to the supervisor.

[0172] When the alarm information indicates that the number of pledged vehicles is consistent, but the frame number information of the pledged vehicles is inconsistent, it can be determined that the IoT device pre-installed on a pledged vehicle may have an abnormal operation; when the number of pledged vehicles is inconsistent, it can be determined that the pledged vehicle is missing. Therefore, the pledged vehicle supervision method provided in the embodiment of this specification can combine the "visual" and "tactile" supervision of pledged vehicles, improve the intensity and accuracy of supervision, and investigate the abnormalities that occur according to the specific circumstances of the alarm information, avoid hardware failures of IoT devices and avoid asset losses.

[0173] The embodiment of this specification also provides a pledge vehicle supervision system, the system comprising:

[0174] A drone, wherein the drone is equipped with a camera for collecting image data of pledged vehicles stored in the warehouse;

[0175] An IoT gateway device, which is integrated on the drone and is used to communicate with an IoT device pre-installed on the pledged vehicle to collect the frame number information of the pledged vehicle;

[0176] A supervision system is connected to the drone and the IoT gateway device, and is used to supervise the pledged vehicles stored in the warehouse based on the image data fed back by the drone, the frame number information fed back by the IoT gateway device and the pledged vehicle supervision method provided in the above embodiment.

[0177] Based on the above-mentioned pledged vehicle supervision method, the embodiments of this specification also provide a pledged vehicle supervision device. The device may include a system (including a distributed system), software (application), module, component, server, client, etc. that uses the method described in the embodiments of this specification and is combined with necessary implementation hardware. Based on the same innovative concept, the devices in one or more embodiments provided in the embodiments of this specification are as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiments of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0178] like Figure 7 As shown, a pledge vehicle supervision device provided in an embodiment of this specification includes:

[0179] The first acquisition module 71 is used to acquire image data of pledged vehicles stored in the warehouse collected by the drone camera;

[0180] An input module 72, used to input the image data into a pre-trained vehicle detection model to obtain the quantity information of the pledged vehicles output by the vehicle detection model;

[0181] A second acquisition module 73 is used to acquire the frame number information of the pledged vehicle collected by an IoT gateway device, wherein the IoT gateway device is integrated on the drone and is in communication connection with an IoT device pre-installed on the pledged vehicle;

[0182] A comparison module 74, used to compare the quantity information of the pledged vehicles and the frame number information with the pre-stored pledged vehicle information respectively;

[0183] The alarm feedback module 75 is used to generate alarm information based on the comparison results and feed it back to the supervisor.

[0184] The beneficial effects obtained by the device provided in the embodiments of this specification are consistent with the beneficial effects obtained by the above method and will not be repeated here.

[0185] like Figure 8As shown, a computer device provided in an embodiment of this specification is provided. The pledged vehicle supervision device in this specification can be a computer device in this embodiment, and the above method of this specification is executed. The computer device 802 may include one or more processors 804, such as one or more central processing units (CPUs), and each processing unit may implement one or more hardware threads. The computer device 802 may also include any memory 806, which is used to store any kind of information such as code, settings, data, etc. Non-limiting, for example, the memory 806 may include any one or more combinations of the following: any type of RAM, any type of ROM, flash memory device, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 802. In one case, when the processor 804 executes an associated instruction stored in any memory or combination of memories, the computer device 802 can perform any operation of the associated instruction. The computer device 802 also includes one or more drive mechanisms 808 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0186] The computer device 802 may also include an input / output module 810 (I / O) for receiving various inputs (via input devices 812) and for providing various outputs (via output devices 814). A specific output mechanism may include a presentation device 816 and an associated graphical user interface (GUI) 818. In other embodiments, the input / output module 810 (I / O), the input device 812, and the output device 814 may not be included, and the computer device 802 may be used as a computer device in a network. The computer device 802 may also include one or more network interfaces 820 for exchanging data with other devices via one or more communication links 822. One or more communication buses 824 couple the components described above together.

[0187] The communication link 822 may be implemented in any manner, for example, through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 822 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc. governed by any protocol or combination of protocols.

[0188] Corresponding to Figures 1 to 6 In addition to the method shown, an embodiment of the specification also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0189] The embodiment of the present specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figures 1 to 6 The method shown.

[0190] The embodiments of the present specification also provide a computer program product, including at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the following Figures 1 to 6 The method shown.

[0191] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0192] It should also be understood that in the embodiments of this specification, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this specification generally indicates that the associated objects before and after are in an "or" relationship.

[0193] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0194] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0195] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or it can be an electrical, mechanical or other form of connection.

[0196] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0197] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0198] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0199] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, according to the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this specification.

Claims

1. A pledge vehicle supervision method, characterized in that: include: Obtain image data of pledged vehicles stored in the warehouse collected by drone cameras; Input the image data into a pre-trained vehicle detection model to obtain the quantity information of the pledged vehicles output by the vehicle detection model; Acquire the vehicle frame number information of the pledged vehicle collected by an IoT gateway device, wherein the IoT gateway device is integrated on the drone and is in communication connection with an IoT device pre-installed on the pledged vehicle; Comparing the quantity information and the frame number information of the pledged vehicles with the pre-stored pledged vehicle information respectively; Based on the comparison results, an alarm message is generated and fed back to the supervisor.

2. The method according to claim 1, characterized in that: Before inputting the image data into a pre-trained vehicle detection model, the method further includes: The image data is spliced.

3. The method according to claim 2, characterized in that The image data is stitched together, further comprising: Sorting the image frames according to the acquisition time of each image frame in the image data; Performing feature point detection on each of the image frames using a feature point algorithm to obtain feature points of each of the image frames; Based on the feature points, feature point matching is performed on any two image frames whose acquisition time is adjacent to each other to obtain a pair of successfully matched feature points; According to the feature point pairs, any two image frames that are adjacent in acquisition time are spliced ​​to obtain spliced ​​image data.

4. The method according to claim 3, characterized in that Performing feature point matching on any two image frames whose acquisition time is adjacent based on the feature point pair to obtain a successfully matched feature point pair further includes: Calculating the distance between any feature point in one of any two image frames adjacent in acquisition time and each feature point in the other of any two image frames adjacent in acquisition time; Determine a first feature point in another image frame that is closest to any one of the feature points; Determine whether the distance between the first feature point and any one of the feature points is less than a preset distance threshold; If yes, determine a second feature point in another image frame that is next closest to the arbitrary feature point, and calculate a distance coefficient between the first feature point and the second feature point; Determining whether the distance coefficient is less than a preset distance coefficient threshold; If yes, determining that the first feature point and any one of the feature points form a feature point pair; Otherwise, it is determined that no feature point matching any one of the feature points exists in the other image frame.

5. The method according to claim 3, characterized in that: According to the feature point pair, any two image frames that are adjacent in acquisition time are spliced ​​to obtain spliced ​​image data, including: Determining, based on the feature point pairs, a homography matrix between any two image frames that are adjacent in acquisition time; According to the homography matrix, any two image frames that are adjacent in acquisition time are spliced ​​to obtain spliced ​​image data.

6. The method according to claim 1, characterized in that The vehicle detection model is trained through the following steps: Acquire vehicle image training data, and label vehicles in the vehicle image training data; Inputting the vehicle image training data into an initial vehicle detection model to obtain a recognition result of the initial vehicle detection model on the vehicle in the vehicle image training data; The initial vehicle detection model is iteratively trained according to the labeling results and the recognition results until the recognition rate of the initial vehicle detection model for the vehicle reaches a preset recognition rate threshold or reaches a preset number of iterative training times, thereby obtaining a trained vehicle detection model.

7. The method according to claim 1, characterized in that The method further comprises: Acquire the target vehicle and pre-defined electronic fence; Determine the position of the image frame where the target vehicle is located in the image data according to the vehicle frame number information; Determining a storage location of the target vehicle in the warehouse according to a location of the image frame where the target vehicle is located in the image data; According to the storage location of the vehicle in the warehouse, determining whether the storage location is located within the electronic fence; If not, an alarm message is generated and fed back to the supervisor.

8. The method according to claim 1, characterized in that Based on the comparison results, an alarm message is generated and fed back to the supervisor, and further: Determine whether the comparison result between the quantity information and the number of vehicles in the pledged vehicle information is consistent; If they are consistent, determining whether the comparison result of the frame number information and the frame numbers of each vehicle in the pledged vehicle information is consistent; If yes, it is determined that there is no abnormality in the pledged vehicle; Otherwise, the alarm information is generated and fed back to the supervisor.

9. A pledge vehicle supervision system, characterized in that: The system comprises: A drone, wherein the drone is equipped with a camera for collecting image data of pledged vehicles stored in the warehouse; An IoT gateway device, which is integrated on the drone and is used to communicate with an IoT device pre-installed on the pledged vehicle to collect the frame number information of the pledged vehicle; A supervision system is connected to the drone and the IoT gateway device, and is used to supervise the pledged vehicles stored in the warehouse based on the image data fed back by the drone, the frame number information fed back by the IoT gateway device, and the pledged vehicle supervision method according to any one of claims 1 to 8.

10. A pledge vehicle supervision device, characterized in that: include: The first acquisition module is used to acquire image data of pledged vehicles stored in the warehouse collected by the drone camera; An input module, used to input the image data into a pre-trained vehicle detection model to obtain the quantity information of the pledged vehicles output by the vehicle detection model; A second acquisition module is used to acquire the frame number information of the pledged vehicle collected by an IoT gateway device, wherein the IoT gateway device is integrated on the drone and is in communication connection with an IoT device pre-installed on the pledged vehicle; A comparison module, used to compare the quantity information of the pledged vehicles and the frame number information with the pre-stored pledged vehicle information respectively; The alarm feedback module is used to generate alarm information based on the comparison results and feed it back to the supervisor.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

13. A computer program product, characterized in that The method comprises at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the method according to any one of claims 1 to 8.

Images