Image processing method, device, computer equipment and computer readable storage medium

Abstract

The present invention discloses an image processing method, apparatus, computer device, and computer-readable storage medium. The method comprises: acquiring an image set; determining, from the image set, a first subset of images belonging to the same moving object and a remaining subset of images excluding the first subset; determining object features of the same moving object based on the first subset of images; searching for a supplementary image from the remaining subset of images based on the object features; and obtaining a trajectory of the same moving object based on the first subset of images and the supplementary image. The present invention addresses the technical problem in related art of generating moving object trajectories, which results in low accuracy.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to an image processing method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] Restoring the trajectory of moving objects is a critical application. For example, in traffic scenarios involving vehicle trajectory restoration, it has important applications in detecting illegal driving behavior, tracking vehicles of key individuals, analyzing traffic conditions, and strengthening public security management. In recent years, a large number of cameras, such as checkpoints and electric police cameras, have been installed on public roads. These cameras generally include sensing capabilities such as vehicle detection and license plate recognition, providing a data source for mining and analyzing vehicle trajectories. When a vehicle's license plate is clearly identifiable, the vehicle's trajectory can be directly derived by linking license plates across cameras. However, in practical applications, license plate recognition can be performed with missing license plates or obstructions to vehicles, resulting in inaccurate vehicle trajectories. Similarly, in scenarios involving restoring moving object trajectories, inaccuracies in the restoration of moving object trajectories can also be a problem.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] Embodiments of the present invention provide an image processing method, apparatus, computer device, and computer-readable storage medium to at least solve the technical problem in the related art of generating a trajectory of a moving object, which results in low accuracy of the obtained trajectory of the moving object.

[0005] According to one aspect of an embodiment of the present invention, there is provided an image processing method, comprising: acquiring an image set; determining from the image set a first image subset belonging to the same moving object, and a remaining image subset excluding the first image subset; determining object features of the same moving object based on the first image subset; searching for a supplementary image from the remaining image subset based on the object features; and obtaining a trajectory of the same moving object based on the first image subset and the supplementary image.

[0006] Optionally, searching for the supplementary image from the remaining image subset according to the object feature includes: determining a range to be searched; and searching for the supplementary image from the remaining image subset based on the range according to the object feature.

[0007] Optionally, determining the range to be searched includes: sorting the images in the first image subset according to shooting time to obtain an initial trajectory, wherein the points in the initial trajectory correspond to the images in the first image subset; dividing the initial trajectory to obtain multiple trajectory segments, determining the spatiotemporal search range corresponding to each trajectory segment, and using the spatiotemporal search range as the range to be searched.

[0008] Optionally, the spatiotemporal search range corresponding to each trajectory segment is determined separately, including: determining the spatial search range of the trajectory segment based on the position corresponding to the starting point and the position corresponding to the ending point of the trajectory segment; determining the temporal search range of the trajectory segment based on the time interval between the starting point and the ending point of the trajectory segment; wherein the spatiotemporal search range includes the spatial search range and the temporal search range.

[0009] Optionally, determining the object features of the same moving object based on the first image subset includes: using a re-recognition model to identify the images included in the first image subset to obtain the features of the images in the first image subset; averaging the features of the images in the first image subset to obtain average features, and using the average features as the object features of the same moving object.

[0010] Optionally, based on the object features, a supplementary image is searched out from the remaining image subset, including: determining a candidate image from the remaining image subset based on the object features; using a verification model to verify the candidate image to obtain a verification result; when the verification result is that the candidate image belongs to the same moving object, determining that the candidate image is the supplementary image.

[0011] Optionally, determining candidate images from the remaining image subset based on the object features includes: obtaining a first eigenvector of the image in the remaining image subset, and a second eigenvector of the object features; determining the distance between the first eigenvector and the second eigenvector; sorting the distances corresponding to multiple first eigenvectors, determining a predetermined number of first eigenvectors with the smallest distances, and determining the images corresponding to the predetermined number of first eigenvectors as the candidate images.

[0012] Optionally, before using the verification model to verify the candidate image and obtaining the verification result, it also includes: obtaining a training sample set, wherein the training sample set includes a positive sample set and a negative sample set, the positive sample set includes samples with the following characteristics: the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are the same moving object, and the negative sample set includes samples with the following characteristics: the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are not the same moving object; training the training sample set to obtain the verification model.

[0013] Optionally, the mobile object includes at least one of the following: a vehicle, a robot, a pet, and an airplane.

[0014] According to another aspect of an embodiment of the present invention, an image processing method is also provided, including: receiving a trajectory generation instruction on an interactive interface; receiving a first image in response to the trajectory generation instruction, wherein the first image corresponds to a predetermined moving object; displaying a first trajectory corresponding to the predetermined moving object on the interactive interface, wherein the first trajectory is generated based on a first image subset, and the first image subset is a set of images in an image set that belong to the same predetermined moving object; displaying a supplementary image on the interactive interface, wherein the supplementary image is an image searched from a remaining image subset excluding the first image subset based on object features of the predetermined moving object, and the object features are determined based on the first image subset; and displaying a synthesized trajectory on the interactive interface, wherein the synthesized trajectory is synthesized based on the first trajectory and the supplementary image.

[0015] According to another aspect of an embodiment of the present invention, a method for image processing of a smart car is provided, including: receiving a first instruction requesting a trajectory of a smart car through a control panel of the smart car; displaying a first trajectory on a display panel of the smart car in response to the first instruction, wherein the first trajectory is generated based on a first image subset, and the first image subset is a set of images in an image set that belong to the same smart car; receiving a second instruction requesting a trajectory of a smart car through the control panel of the smart car; and displaying a second trajectory of the same smart car on the display panel of the smart car in response to the second instruction, wherein the second trajectory is generated based on the first image subset and a supplementary image, and the supplementary image is searched from a remaining image subset based on smart car features, the smart car features are determined based on the first image subset, and the remaining image subset is a set of images in the image set excluding the first image subset.

[0016] According to another aspect of an embodiment of the present invention, an image processing device is also provided, including: a first acquisition module for acquiring an image set; a first determination module for determining a first image subset belonging to the same moving object and a remaining image subset excluding the first image subset from the image set; a second determination module for determining object features of the same moving object based on the first image subset; a search module for searching for a supplementary image from the remaining image subset based on the object features; and a processing module for obtaining a trajectory of the same moving object based on the first image subset and the supplementary image.

[0017] According to another aspect of an embodiment of the present invention, an image processing device is further provided, including: a first receiving module for receiving a trajectory generation instruction on an interactive interface; a second receiving module for receiving a first image in response to the trajectory generation instruction, wherein the first image corresponds to a predetermined moving object; a first display module for displaying a first trajectory corresponding to the predetermined moving object on the interactive interface, wherein the first trajectory is generated based on a first image subset, and the first image subset is a set of images in an image set that belong to the same predetermined moving object; a second display module for displaying a supplementary image on the interactive interface, wherein the supplementary image is an image searched from a remaining image subset excluding the first image subset based on object features of the predetermined moving object, and the object features are determined based on the first image subset; and a third display module for displaying a synthesized trajectory on the interactive interface, wherein the synthesized trajectory is synthesized based on the first trajectory and the supplementary image.

[0018] According to another aspect of an embodiment of the present invention, an image processing device for a smart car is further provided, including: a third receiving module, configured to receive a first instruction requesting a trajectory of a smart car through a control panel of the smart car; a first responding module, configured to display a first trajectory on a display panel of the smart car in response to the first instruction, wherein the first trajectory is generated based on a first image subset, which is a set of images in an image set that belong to the same smart car; a fourth receiving module, configured to receive a second instruction requesting a trajectory of a smart car through the control panel of the smart car; a second responding module, configured to display a second trajectory of the same smart car on the display panel of the smart car in response to the second instruction, wherein the second trajectory is generated based on the first image subset and a supplementary image, wherein the supplementary image is searched from a remaining image subset based on smart car features, wherein the smart car features are determined based on the first image subset, and the remaining image subset is a set of images in the image set excluding the first image subset.

[0019] According to another aspect of an embodiment of the present invention, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program; and the processor is configured to execute the computer program stored in the memory, wherein when the computer program is executed, the processor executes any one of the image processing methods.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform any one of the image processing methods described above.

[0021] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, any one of the image processing methods described above is implemented.

[0022] In an embodiment of the present invention, a supplementary image is searched based on object features, and the trajectory of the same moving object is obtained through the first image subset and the supplementary image. Since the trajectory of the moving object is also obtained based on the supplementary image, the obtained trajectory of the moving object is more accurate than that obtained based only on the first image subset, thereby achieving the technical effect of obtaining a more accurate trajectory of the moving object, and further solving the technical problem in the related art of generating the trajectory of the moving object, which results in low accuracy of the obtained trajectory of the moving object. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 A hardware structure block diagram of a computer terminal for implementing an image processing method is shown;

[0025] Figure 2 is a flowchart of an image processing method 1 according to embodiment 1 of the present invention;

[0026] Figure 3 is a flowchart of a second image processing method according to embodiment 1 of the present invention;

[0027] Figure 4 is a flowchart of an image processing method for an intelligent vehicle according to embodiment 1 of the present invention;

[0028] Figure 5 is a flow chart of a vehicle trajectory restoration method provided according to an optional embodiment of the present invention;

[0029] Figure 6is a structural block diagram of an image processing device 1 provided according to embodiment 2 of the present invention;

[0030] Figure 7 is a structural block diagram of an image processing device 2 provided according to embodiment 3 of the present invention;

[0031] Figure 8 2 is a block diagram of a structure of an image processing device for an intelligent vehicle according to a fourth embodiment of the present invention;

[0032] Figure 9 is a device block diagram of a terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0036] Initial trajectory restoration (observed trajectory): The trajectory is directly restored by license plate association.

[0037] Secondary trajectory restoration: Based on the initial trajectory restoration, a finer-grained trajectory is generated through further completion.

[0038] Example 1

[0039] According to an embodiment of the present invention, an image processing method embodiment is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0041] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0042] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the image processing method of the application described above. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0044] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0045] Under the above operating environment, this application provides Figure 2 The image processing method shown. Figure 2 is a flow chart of an image processing method 1 according to embodiment 1 of the present invention. Figure 2 As shown, the method includes the following steps:

[0046] Step S202, obtaining an image set;

[0047] Step S204, determining a first image subset belonging to the same moving object and remaining image subsets excluding the first image subset from the image set;

[0048] Step S206, determining object features of the same moving object based on the first image subset;

[0049] Step S208, searching for a supplementary image from the remaining image subset based on the object features;

[0050] Step S210 : obtaining a trajectory of the same moving object based on the first image subset and the supplementary image.

[0051] Through the above steps, the supplementary image is searched according to the object features, and the trajectory of the same moving object is obtained through the first image subset and the supplementary image. Since the trajectory of the moving object is also obtained based on the supplementary image, the obtained trajectory of the moving object is more accurate than that obtained based only on the first image subset, thereby achieving the technical effect of obtaining a more accurate trajectory of the moving object, and further solving the technical problem in the related art of generating the trajectory of the moving object, which will result in low accuracy of the obtained trajectory of the moving object.

[0052] As an optional embodiment, the moving object can be of various types, such as vehicles, robots, pets, airplanes, and so on. The above-mentioned moving objects can generate trajectories in actual application scenarios. In different application scenarios, different moving objects can be selected to obtain the trajectories of the above-mentioned moving objects. For example, when the moving object is a vehicle, it can be in a traffic scene where the vehicle trajectory is restored, in a traffic scene where the vehicle is tracked, and so on; when the moving object is a robot, it can be in an experimental scene where the robot trajectory is restored, in a scene where the robot automatically tracks other objects, and so on; when the moving object is a pet, it can be in an actual life scene where the pet trajectory is tracked, in an experimental scene where the animal habits are detected, and so on; when the moving object is an airplane, it can be in a scene where the airplane's operating route is detected, and so on. Selecting different moving objects for different scenarios to solve the problems targeted by each scenario is of great significance for applications involving trajectory detection.

[0053] As an optional embodiment, the image can be of various types, including images of various moving objects. Different types of images can be selected based on different scenarios. For example, in a traffic scenario involving vehicle trajectory restoration, the moving object can be a vehicle, so the image can be an image of a license plate or an image of the vehicle body. The license plate image can be an image of a single-layer license plate or a multi-layered license plate. The vehicle body image can include a wide variety of vehicle types, such as large vehicles (trucks, semi-trailer tractors, trams), trailers, motorcycles, large and medium-sized buses, and so on. It should be noted that the aforementioned image can contain images of multiple license plates or vehicle bodies. Therefore, this optional embodiment can be applied not only to license plate recognition in traffic scenarios involving vehicle trajectory restoration, but also to vehicle body recognition, or images related to other vehicles of different designs, effectively resolving the current technical issue of low accuracy in determining vehicle trajectories. This optional embodiment has no restrictions on license plate type, vehicle body conditions, or the number of license plates or vehicle bodies contained in the image, effectively achieving the versatility of obtaining more accurate vehicle trajectories through images. It can be seen from this optional embodiment that in other application scenarios where moving objects can generate tracks, the use of adaptive images is universal in various scenarios.

[0054] As an optional embodiment, a plurality of methods can be used to obtain an image set. Different moving objects and different application scenarios require different methods for obtaining image sets. For example, when an image set is based on a traffic scene restored from a vehicle trajectory and the moving object is a vehicle, it can be directly obtained by road monitoring shooting, high-speed speed measurement shooting, camera shooting, and the like. For another example, before obtaining an image set, the original image can be obtained first, and the original image can be recognized to identify an image set related to the vehicle. When recognizing the original image, a plurality of methods can be used, such as license plate recognition and vehicle body recognition. By adopting the above-mentioned processing, the obtained image set is related to the vehicle, which can effectively avoid the misidentification of the image, such as identifying an image without a vehicle. In addition, when obtaining the original image, a plurality of implementation methods can also be used, such as the following implementation methods: road monitoring shooting, high-speed speed measurement shooting, camera shooting, and the like. After acquiring the original image, before identifying the image and acquiring the image set, the acquired original image may also be preprocessed. For example, it may include at least one of the following processing: adjusting the image tone, such as brightness, contrast, etc.; adjusting the image size, such as zooming in and out, adjusting the length, width, and height, etc.; cropping different positions of the image, for example, cropping a specified target object area in the original image. For example, when monitoring vehicle speed on a highway, a front image of a large vehicle is captured. In addition to the license plate and vehicle body, the image also includes multiple objects such as the front and surrounding background. For example, if the original image includes trees, pedestrians, and other vehicles, the original image needs to be identified, the vehicle designated as the target object, and the relevant vehicle image region extracted. The vehicle portion of the original image needs to be cropped. If there are multiple target objects, i.e., multiple vehicles, different vehicles in the original image can be cropped separately and added to an image collection to facilitate subsequent integration of images of the same vehicle to obtain the vehicle's trajectory. If the original image is dark due to cloudy weather, the image brightness can be increased to better identify the image. If the license plate is located low on the front of the vehicle when capturing the image, the image captured by the camera at a high position will be distorted, and the license plate height problem is likely to occur. In this case, the original image can be adjusted to a higher height, so that the final image collection can more clearly reflect the vehicle's trajectory. For example, if the image set is based on a scenario involving detecting aircraft routes and the moving object is an aircraft, it can be directly acquired through satellite photography or other means. For another example, before acquiring the image set, the original images can be acquired and then recognized to identify the image set related to the aircraft. Recognition of the original images can be performed using a variety of methods, such as license plate recognition and aircraft body recognition.When the moving objects are of different types and the application scenarios are different, the appropriate method for obtaining the image set is selected according to the actual situation. I will not go into details here, so that the final image set can reflect the trajectory of the moving object more clearly, realistically and appropriately.

[0055] As an optional embodiment, a first image subset belonging to the same moving object and the remaining image subsets excluding the first image subset are determined from the image set. That is, the images belonging to the same moving object are first divided into the same image subset, and the remaining images that are not clearly identified are divided into the remaining image subsets. There are many ways to determine whether the moving objects in the image belong to the same moving object. For example, in a traffic scene based on vehicle trajectory restoration, when the moving object is a vehicle, it includes license plate recognition, vehicle body recognition, and so on. In the license plate recognition scenario, images with complete license plate recognition digits and high confidence in the recognition results can be divided into vehicle images for vehicle identity confirmation, and then the images belonging to the same moving object in the vehicle identity confirmation images are divided into the same image subset. Therefore, in the license plate recognition scenario, images with incomplete license plate recognition digits or low confidence in the recognition results will be divided into the remaining image subset for subsequent verification. Images identified as the same moving object in an image set can be included in the same image subset. When other standards or conditions can be used to verify that the moving objects shown in the images in the image set are the same moving object, this can also be applied to the present application. Strict identification standards can ensure the accuracy of determining the trajectory of the moving object, thereby effectively ensuring that the images in the same image subset are the same moving object.

[0056] As an optional embodiment, the object features of a moving object are features that can highlight the characteristics of the moving object, that is, the moving object can be identified based on the object features, and can be represented by a feature vector. The object features can be determined based on the moving object, or based on the moving object and the application scenario. For example, when the moving object is a vehicle, the object features can be the license plate, body, or other features of the vehicle; in the scenario of restoring the trajectory of large vehicles, the object features can be the license plate, body, or other features of a truck, semi-trailer tractor, large tram, etc. The above features are represented by a feature vector. The object features of the same moving object are determined based on the first image subset, that is, the object features of the vehicle are determined in the same image subset. When the object features are represented by a feature vector, a query vector for search is obtained based on the feature vector, that is, the object features of the same moving object are determined. There are many ways to obtain feature vectors. For example, the feature vector of the moving object image can be extracted through a re-identification model. When obtaining the feature vector of the moving object, there are also many situations. For example, if the first image subset contains an image of the same moving object, then the feature vector of the image can be calculated and used as the query vector for the search. For example, if the first image subset contains two or more images of the same moving object with the same trajectory segment, then the feature vectors of the moving object images can be calculated and averaged, and the feature vector of the trajectory segment with the average value can be used as the query vector for the search. For example, if the first image subset contains two or more images of the same moving object with different trajectory segments, then the feature vectors of each moving object image in each trajectory segment of the image can be calculated and averaged, and the feature vector of the trajectory segment with the average value can be used as the query vector for the search. The feature vector is typically a 256- or 512-dimensional real-valued vector. Generally speaking, if the first image subset contains multiple images of the same moving object, then a re-identification model is used to identify the images included in the first image subset to obtain features of the images in the first image subset. The features of the images in the first image subset are averaged to obtain average features, and the average features are used as the object features of the same moving object. The object feature is used as a search query vector, and the search query vector is used as a condition to perform a subsequent search to verify whether it is the same moving object.

[0057] As an optional embodiment, supplementary images are searched for from the remaining image subset based on object features, including various methods, such as determining a range to be searched; and searching for supplementary images from the remaining image subset based on the range based on the object features. When determining the range to be searched, the search range can be determined based on time, spatial location, or both time and spatial location. Before determining the range to be searched, the images in the first image subset can be sorted based on capture time to obtain an initial trajectory, i.e., an initial trajectory result for the same moving object. The search range can then be divided based on the initial trajectory. For example, if the initial trajectory is long, the initial trajectory can be divided based on points in the initial trajectory to obtain multiple trajectory segments, where the points in the initial trajectory correspond to different images of the same moving object in the first image subset. It should be noted that when dividing the initial trajectory, the initial trajectory can be divided based on point locations, i.e., based on the camera's capture point location, and the spatiotemporal search range corresponding to each trajectory segment can be determined separately, and the spatiotemporal search range can be used as the range to be searched. Alternatively, the initial trajectory can be divided based on other conditions. It can reduce the computational complexity of identifying moving object images when searching for supplementary images, and make the recognition range more accurate and detailed, which is conducive to improving the efficiency of searching for supplementary images, so that the most appropriate range can be selected for search based on the scene and actual needs.

[0058] As an optional embodiment, determining the spatiotemporal search range corresponding to each trajectory segment includes: determining the spatial search range of the trajectory segment based on the position corresponding to the starting point and the position corresponding to the ending point of the trajectory segment; and determining the temporal search range of the trajectory segment based on the time interval between the starting point and the ending point of the trajectory segment. The spatiotemporal search range includes a spatial search range and a temporal search range. Searching based on the temporal and spatial ranges ensures the spatiotemporal validity of the search. Searching within the same area and time period avoids searching a large area all the time, reduces computational effort, and improves image processing efficiency.

[0059] It should be noted that when determining the spatial search range of the trajectory segment based on the position corresponding to the starting point of the trajectory segment and the position corresponding to the ending point, the spatial search range of the trajectory segment can be determined directly based on the distance between the position corresponding to the starting point of the trajectory segment and the position corresponding to the ending point. For example, the distance between the position corresponding to the starting point of the trajectory segment and the position corresponding to the ending point is taken as the diameter, and the circular area obtained is the spatial search range. It should be pointed out that the distance between the position corresponding to the starting point of the trajectory segment and the position corresponding to the ending point is used as the diameter here for only one example, and this processing can be for the case where the trajectory is relatively smooth. When the trajectory change is relatively large, the maximum distance between the positions included between the position corresponding to the starting point of the trajectory segment and the position corresponding to the ending point can be determined as the diameter, and the circular area covered by the diameter is determined as the spatial search range.

[0060] As an optional embodiment, when searching for supplementary images from the remaining image subset based on object features, the following method can also be used: candidate images are determined from the remaining image subset based on the object features; the candidate images are verified using a verification model to obtain a verification result; and if the verification result indicates that the candidate images belong to the same moving object, the candidate image is determined to be a supplementary image. This allows the remaining image subset to be fully utilized, and the moving objects in the image subset can be grouped into corresponding moving object trajectories. Using the verification model ensures that candidate images obtained from the remaining image subset can be grouped into corresponding moving object trajectories with high accuracy. During the verification process of the verification model, a comparison can be performed between the first image subset and the candidate images to determine whether they belong to the same moving object. Specifically, for each candidate image obtained from the search, its attributes, such as the re-identification features, the location of the passing points, and the elapsed time, are input into a verification model, and a binary classification result is output. If the classification result is "yes," it indicates that they belong to the same moving object; if the classification result is "no," it indicates that they do not belong to the same moving object. Candidate images belonging to the same moving object are retained as supplementary images, while candidate images that do not belong to the same moving object are deleted. It is ensured that the retained candidate image, that is, the supplementary image, is the same moving object as the moving object in the first image subset, thereby ensuring the accuracy of the trajectory. Moreover, by using the supplementary image to restore the moving object trajectory, the restored trajectory image can be ensured to be more comprehensive and accurate.

[0061] As an optional embodiment, before verifying a candidate image using a verification model and obtaining a verification result, the method further includes: obtaining a training sample set, wherein the training sample set includes a positive sample set and a negative sample set, the positive sample set including samples having the following characteristics: the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are the same moving object, and the negative sample set including samples having the following characteristics: the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are not the same moving object; and training the training sample set to obtain a verification model. The verification model obtained by training the training sample set using positive and negative samples effectively avoids inaccurate recognition and inaccurate determination of moving object trajectories because the training samples include positive samples in which the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are the same moving object, and negative samples in which the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are not the same moving object. This allows the verification model used for the candidate image to better learn the features that distinguish the positive and negative samples, enabling a more robust judgment of whether the moving object is the same moving object. Continuous sample training improves the accuracy of the verification model and gradually improves the accuracy of the moving object trajectory.

[0062] As an optional embodiment, the verification model may include multiple types, for example, a verification model based on a deep neural network. It should be noted that the verification model based on a deep neural network mentioned above is only an example, and other verification models not listed one by one can also be applied to this application. Through training, the candidate image is verified. It can be seen that the verification model can be based on different mechanisms and can be selected according to different needs, providing a variety of methods to choose from, making it more flexible and convenient to use, and greatly improving the applicability of candidate image verification.

[0063] As an optional embodiment, there are many ways to determine candidate images from the remaining image subset based on object features. For example, the following method can be used: obtain the first feature vector of each image in the remaining image subset and the second feature vector of the object feature; determine the distance between the first feature vector and the second feature vector; sort the distances corresponding to the multiple first feature vectors, determine a predetermined number of first feature vectors with the smallest distances, and determine the images corresponding to the predetermined number of first feature vectors as candidate images. For example, the remaining image subset can be searched using the query vector and the search spatiotemporal range as query conditions. Specifically, for each image in the remaining image subset within the search spatiotemporal range, its re-identified feature vector is extracted, and the Euclidean distance between the re-identified feature vector and the query vector is calculated. The images are sorted according to the Euclidean distance, and the first several moving object images with the smallest distances are returned, i.e., images with a high accuracy and a high similarity rate. It should be noted that the predetermined number of moving object images returned can be preset, for example, by a pre-defined method.

[0064] As an optional embodiment, the verification result can also be used in a variety of scenarios. For example, when the moving object is a vehicle in a scenario based on vehicle trajectory restoration, the verification result can also be used for at least one of the following: locating the vehicle marked by the verification result; measuring the speed of the vehicle marked by the verification result; monitoring the vehicle marked by the verification result. The verification result can be used in a variety of scenarios and for a variety of purposes, such as: vehicle trajectory query and mining, vehicle speeding measurement, vehicle search and comparison, illegal driving behavior, tracking key personnel and vehicles, analyzing traffic conditions, strengthening public security management, etc. The ability to restore vehicle trajectories is an extremely important and basic capability. The accuracy of vehicle trajectory restoration is related to many subsequent applications. Accurate identification of the vehicle is guaranteed by accurate secondary trajectory restoration. When the moving object is other, the same effect can be achieved, the functional application in different scenarios can be enhanced, and multiple purposes can be achieved more flexibly and accurately.

[0065] As an optional embodiment, based on the first image subset and the supplementary image, the trajectory of the same moving object is obtained, and the first image subset and the supplementary image are sorted in chronological order to obtain a secondary trajectory of the same moving object, so that the trajectory image of the same moving object is complete, and the trajectory is restored secondary. On the basis of the initial trajectory restoration, a finer-grained trajectory is generated by further completion. Through the secondary trajectory restoration, the moving object image that is directly associated with the moving object and ignored can be added, thereby obtaining a finer-grained moving object trajectory restoration result.

[0066] Figure 3 is a flow chart of the second image processing method according to embodiment 1 of the present invention. Figure 3 As shown, the method includes the following steps:

[0067] Step S302, receiving a trajectory generation instruction on the interactive interface;

[0068] Step S304, in response to the trajectory generation instruction, receiving a first image, wherein the first image corresponds to a predetermined moving object;

[0069] Step S306 , displaying a first trajectory corresponding to the predetermined moving object on the interactive interface, wherein the first trajectory is generated based on a first image subset, and the first image subset is a set of images in the image set that belong to the same predetermined moving object;

[0070] Step S308, displaying a supplementary image on the interactive interface, wherein the supplementary image is an image searched from the remaining image subsets excluding the first image subset based on the object features of the predetermined moving object, and the object features are determined based on the first image subset;

[0071] Step S310 : Displaying a synthesized trajectory on an interactive interface, wherein the synthesized trajectory is synthesized based on the first trajectory and the supplementary image.

[0072] Through the above steps, the supplementary image is searched based on object features, and the trajectory of the same moving object is obtained using the first image subset and the supplementary image. Because the moving object trajectory is also obtained based on the supplementary image, the obtained moving object trajectory is more accurate than that obtained based on the first image subset alone, thereby achieving the technical effect of obtaining a more accurate moving object trajectory. This solves the technical problem of low accuracy in generating moving object trajectories in related technologies. In addition, by displaying the above-mentioned first trajectory, supplementary image, and synthesized trajectory on the interactive interface, users can more intuitively and accurately obtain the moving object trajectory, further improving the user experience.

[0073] Figure 4 FIG. 1 is a flow chart of an image processing method for an intelligent vehicle according to embodiment 1 of the present invention. Figure 4 As shown, the method includes the following steps:

[0074] Step S402: receiving a first instruction requesting a track of the smart car through a control panel of the smart car;

[0075] Step S404: Displaying a first trajectory on a display panel of the smart car in response to the first instruction, wherein the first trajectory is generated based on a first image subset, and the first image subset is a set of images in the image set that belong to the same smart car;

[0076] Step S406, receiving a second instruction requesting the trajectory of the smart car through the control panel of the smart car;

[0077] Step S408: In response to the second instruction, a second trajectory of the same smart car is displayed on the display panel of the smart car, wherein the second trajectory is generated based on the first image subset and the supplementary image, the supplementary image is searched from the remaining image subset based on the smart car characteristics, the smart car characteristics are determined based on the first image subset, and the remaining image subset is the set of images in the image set excluding the first image subset.

[0078] Through the above steps, a supplementary image is searched based on the smart car's features, and the trajectory of the same smart car is obtained using the first image subset and the supplementary image. Since the smart car trajectory is also obtained based on the supplementary image, the obtained smart car trajectory is more accurate than that obtained based solely on the first image subset, thereby achieving the technical effect of obtaining a more accurate smart car trajectory, thereby resolving the technical problem of low accuracy in generating smart car trajectories in related technologies. In addition, by displaying the above-mentioned first and second trajectories on the control panel of the smart car, not only can the user obtain the smart car trajectory more intuitively and accurately, but by displaying the first and second trajectories separately, the difference between the two can be effectively demonstrated, thereby allowing the user to clearly feel the significant effect of adding the supplementary image, effectively improving the user experience.

[0079] Based on the above embodiment and optional embodiment, an optional implementation manner is provided. In this optional implementation manner, in a scenario where a vehicle restores a trajectory, a vehicle is used as an example of a moving object, which is described in detail below.

[0080] In related technologies, when a vehicle's license plate is clearly identifiable, the vehicle's trajectory can be directly derived by linking license plates across cameras. However, in practical applications, license plate recognition often involves missing license plates, mutual occlusion between vehicles, and environmental factors such as weather and lighting that affect license plate image quality. These factors can lead to inaccurate license plate recognition results, preventing the vehicle's true identity from being fully identified, resulting in the loss of the vehicle's fine-grained trajectory and, consequently, low accuracy in vehicle trajectory restoration. Furthermore, human actions such as occlusion, forgery, and license plate swapping can obscure the vehicle's true trajectory, misleading law enforcement officers and reducing the accuracy of vehicle trajectory restoration. Therefore, it is necessary to accurately and maximally granularly restore the vehicle's true trajectory.

[0081] Based on this, in an optional embodiment of the present invention, a vehicle secondary trajectory restoration method based on integrated search and reasoning is provided. Through secondary trajectory restoration, the license plate image that is ignored by the direct license plate association can be added to obtain a more fine-grained vehicle trajectory restoration result. Figure 5 is a flow chart of a vehicle trajectory restoration method provided according to an optional embodiment of the present invention. Figure 5 The following describes in detail the optional implementation of the present invention.

[0082] S1, vehicle image database partition;

[0083] Vehicle images are divided into a library of vehicle images with confirmed vehicle identities and a library of vehicle images with undetermined vehicle identities. Specifically, images with complete and accurate vehicle recognition results (i.e., images with a complete license plate recognition digit count and high confidence) are classified into the library of vehicle images with confirmed vehicle identities, while the remaining images are classified into the library of vehicle images with undetermined vehicle identities.

[0084] S2, initial vehicle trajectory restoration;

[0085] In the vehicle identity image database, vehicle images with identical license plates are arranged in chronological order to form a track. Specifically, based on the license plate recognition results, vehicle images with identical license plate recognition results are categorized and then arranged into a track based on the time these images were captured. This track constitutes the initial vehicle track restoration. It should be noted that an observed track includes multiple observed points. Each camera point in an observed track is called an observed point, and the number of camera points included in the observed track is denoted as the track length.

[0086] S3, search phase;

[0087] Perform an image search on potential image areas in a vehicle image library where the vehicle identity is not determined, and return a batch of images to be verified. Specifically, the process also includes the following steps:

[0088] 1) Select the search query vector. For each initially restored trajectory, the vehicle re-ID model is first used to extract a feature vector (typically a 256- or 512-dimensional real-valued vector) for each vehicle image. The average of the feature vectors of all vehicle images in each trajectory is taken as the feature vector for that vehicle, and this vector is used as the query vector for the search.

[0089] 2) Determine the search's spatiotemporal scope. For each segment of the initially restored trajectory, determine the search's spatiotemporal scope. Specifically, for a trajectory of length L, determine a spatiotemporal search scope of L-1 segments. The spatiotemporal search scope is constructed as follows: the search's spatial scope is defined as a circular region with a diameter defined by each pair of adjacent observed points, and the search's temporal scope is defined as the time interval between each pair of adjacent observed points.

[0090] 3) Search process. Using the query vector from step 1) and the search spatiotemporal range determined in step 2) as query conditions, a search is performed in the vehicle image database. Specifically, for each image in the vehicle image database that falls within the search spatiotemporal range, its vehicle re-identification feature vector is extracted, and the Euclidean distance between this vector and the query vector is calculated. The images are sorted by Euclidean distance, and the first several (a pre-defined number, for example, can be specified) vehicle images with the smallest distance are returned.

[0091] S4, verification phase;

[0092] Verify one by one whether the images searched back in step S3 and the initially restored trajectory belong to the same vehicle. Keep the images that belong to the same vehicle, and delete the images that do not belong to the same vehicle. Specifically, use a vehicle identity verification model. Compare and confirm the vehicle identity of the observed trajectory in S2 and the identity of the image returned by S3 to determine whether they belong to the same vehicle. Specifically, for each image returned by the search, input its and the vehicle re-identification features, passing point locations, passing time and other attributes of the image of the observed trajectory into a verification model, and output a binary classification result. If the classification result is "yes", it means it belongs to the same vehicle; if the classification result is "no", it means it does not belong to the same vehicle. Keep the images that belong to the same vehicle, and delete the images that do not belong to the same vehicle.

[0093] S5, trajectory merging.

[0094] The image retained by S4 and the vehicle trajectory of S2 are combined to form a secondary trajectory restoration result. Specifically, the image retained by S4 and the vehicle trajectory of S2 are combined, and all images are arranged in chronological order to output the secondary trajectory restoration result.

[0095] Through the above steps, the vehicle trajectory restoration method based on integrated search and reasoning in an optional embodiment of the present invention can be implemented. Taking actual vehicle trajectory restoration as an example, a detailed description is given below.

[0096] S1. Vehicle image database partitioning. Approximately 5 million vehicle images captured within a single day by N = 100 camera locations in key areas of a city were collected. Approximately 4 million of these images had complete and accurate vehicle recognition results and were collected into a vehicle image database with confirmed vehicle identities. The remaining images, whose identities were unrecognizable, were collected into a vehicle image database with uncertain vehicle identities. For the vehicle image database with uncertain vehicle identities, a search engine was established using ES (Elastic Search), which supports indexing of capture time and points passed through, and returns search results sorted by feature vector distance.

[0097] In step S2, vehicles in the identified vehicle image database are categorized by license plate. Images of the same category are sorted in chronological order to form the initial trajectory restoration results. This results in approximately 400,000 trajectories, with an average length of 10.

[0098] In S3, a trained residual network (ResNet), such as ResNet-50, is used as a vehicle image feature extraction model, with the model output being 512-dimensional features. For each observed trajectory, features from each image are extracted and averaged as the query feature for that trajectory. A spatial and temporal search range is determined for each observed trajectory. Within the library of vehicle images with uncertain identities, a search is performed based on the search range in both space and time. For each pair of adjacent observed points, the top 10 images with the closest distance are returned.

[0099] In step S4, for each image returned by the search, its attributes, such as vehicle re-identification features, location of passing points, and time of passing, are sequentially input into a verification model. This model is a Long Short-Term Memory (LSTM) network, with the number of LSTM cycles equal to the trajectory length. The point input is a one-hot vector of N = 100 dimensions. The time input is a 24-dimensional time vector. The specific representation of the time vector is as follows: Assuming the hour and minute format of time is h:m (0 <= h <= 23, 0 <= m <= 59), the vector values at the hth (vector position index starts at 0) and h+1th positions are 1-m / 60 and m / 60, respectively, and the remaining 22 positions are all 0. For example, the vector representation for 02:15 is . The output is the binary classification result.

[0100] It should be noted that the validation model is trained using a common deep neural network training method. Prior to training, a batch of data must be collected as the validation model training set. This training set must contain an equal number of positive samples (the observed trajectory and the image to be validated are the same vehicle) and negative samples (the observed trajectory and the image to be validated are different vehicles). Trajectories within the same training batch must have the same length. The training optimizer used is stochastic gradient descent (SGD). The training parameters are as follows: Momentum SGD optimizer with a momentum of 0.9 is used. Weight decay is set to 5e-4. The initial learning rate is 0.1, and then decays by 1 / 10 every 10 training epochs. Each batch contains 64 samples, and the model is trained 50 epochs in total. The trained validation model is then used for testing. For each observed trajectory and the image to be validated returned by the search, the validation model is fed into the validation model, which then outputs the classification result. If the classification result is "yes", it means they belong to the same vehicle; if the classification result is "no", it means they do not belong to the same vehicle. Images belonging to the same vehicle are retained, and images that do not belong to the same vehicle are deleted.

[0101] S5, merges the image retained by S4 and the vehicle trajectory of S2, arranges all images in chronological order, and outputs the secondary restoration result of the trajectory.

[0102] Through the above optional implementation, the following beneficial effects can be achieved:

[0103] (1) Through the integrated search and reasoning framework, the secondary restoration of vehicle trajectories is achieved. By adding license plate images that are ignored by the direct license plate association, a more refined vehicle trajectory restoration result can be obtained;

[0104] (2) A scalable and flexible framework is constructed, in which the vehicle re-ID feature extraction, search, and reasoning parts can be updated to advanced methods to further improve the accuracy of trajectory restoration.

[0105] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0106] Through the description of the above embodiments, those skilled in the art can clearly understand that the image processing method according to the above embodiments can be implemented by means of software plus the necessary general hardware platform, or of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0107] Example 2

[0108] According to an embodiment of the present invention, a device 1 for implementing the above-mentioned image processing method 1 is also provided. FIG. 1 is a structural block diagram of the image processing device 1 provided according to embodiment 2 of the present invention. Figure 6 As shown, the device includes: a first acquisition module 602, a first determination module 604, a second determination module 606, a search module 608 and a processing module 610. The device is described below.

[0109] A first acquisition module 602 is used to acquire an image set; a first determination module 604 is connected to the above-mentioned first acquisition module 602, and is used to determine a first image subset belonging to the same moving object, and a remaining image subset excluding the first image subset, from the image set; a second determination module 606 is connected to the above-mentioned first determination module 604, and is used to determine the object features of the same moving object based on the first image subset; a search module 608 is connected to the above-mentioned second determination module 606, and is used to search for a supplementary image from the remaining image subset based on the object features; a processing module 610 is connected to the above-mentioned search module 608, and is used to obtain the trajectory of the same moving object based on the first image subset and the supplementary image.

[0110] It should be noted that the first acquisition module 602, the first determination module 604, the second determination module 606, the search module 608, and the processing module 610 described above correspond to steps S202 to S210 in Example 1. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0111] Example 3

[0112] According to an embodiment of the present invention, a second apparatus for implementing the above-mentioned second image processing method is also provided. Figure 7 : is a structural block diagram of an image processing device 2 provided according to embodiment 3 of the present invention, as shown in FIG. Figure 7 As shown, the device includes: a first receiving module 702, a second receiving module 704, a first display module 706, a second display module 708 and a third display module 710. The device is described below.

[0113] A first receiving module 702 is configured to receive a trajectory generation instruction on an interactive interface; a second receiving module 704 is connected to the first receiving module 702 and is configured to respond to the trajectory generation instruction and receive a first image, wherein the first image corresponds to a predetermined moving object; a first display module 706 is connected to the second receiving module 704 and is configured to display a first trajectory corresponding to the predetermined moving object on the interactive interface, wherein the first trajectory is generated based on a first image subset, and the first image subset is a set of images in an image set that belong to the same predetermined moving object; a second display module 708 is connected to the first display module 706 and is configured to display a supplementary image on the interactive interface, wherein the supplementary image is an image searched from the remaining image subsets excluding the first image subset based on object features of the predetermined moving object, and the object features are determined based on the first image subset; a third display module 710 is connected to the second display module 708 and is configured to display a synthesized trajectory on the interactive interface, wherein the synthesized trajectory is synthesized based on the first trajectory and the supplementary image.

[0114] It should be noted that the first receiving module 702, the second receiving module 704, the first display module 706, the second display module 708, and the third display module 710 described above correspond to steps S302 to S310 in Example 1. The examples and application scenarios implemented by the modules and corresponding steps are the same, but are not limited to those disclosed in Example 1. It should be noted that the modules described above, as part of the apparatus, can be run in the computer terminal 10 provided in Example 1.

[0115] Example 4

[0116] According to an embodiment of the present invention, a device for implementing the above-mentioned image processing method for a smart car is also provided. Figure 8 FIG. 4 is a block diagram of an image processing device for an intelligent vehicle according to a fourth embodiment of the present invention. Figure 8 As shown, the device includes: a third receiving module 802, a first responding module 804, a fourth receiving module 806 and a second responding module 808. The device is described below.

[0117] A third receiving module 802 is configured to receive, via the control panel of the smart car, a first instruction requesting a trajectory of the smart car; a first responding module 804 is connected to the third receiving module 802 and configured to respond to the first instruction by displaying a first trajectory on the display panel of the smart car, wherein the first trajectory is generated based on a first image subset, which is a set of images in the image set that belong to the same smart car; a fourth receiving module 806 is connected to the first responding module 804 and configured to receive, via the control panel of the smart car, a second instruction requesting a trajectory of the smart car; a second responding module 808 is connected to the fourth receiving module 806 and configured to respond to the second instruction by displaying a second trajectory of the same smart car on the display panel of the smart car, wherein the second trajectory is generated based on the first image subset and a supplementary image, wherein the supplementary image is searched from the remaining image subset based on the smart car features, the smart car features being determined based on the first image subset, and the remaining image subset being a set of images in the image set excluding the first image subset.

[0118] It should be noted that the third receiving module 802, the first response module 804, the fourth receiving module 806, and the second response module 808 correspond to steps S402 to S408 in Example 1. The examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the modules described above, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0119] Example 5

[0120] The embodiment of the present invention can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0121] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0122] Optionally, Figure 9 1 is a block diagram of a computer terminal according to an embodiment of the present invention. Figure 9 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 91, and a memory 92 for storing processor-executable instructions; wherein the processor is configured to execute instructions to implement any of the above-mentioned image processing methods.

[0123] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image processing method and device in the embodiments of the present invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the above-mentioned image processing method. The memory may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include a memory remotely located relative to the processor, and these remote memories may be connected to terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain an image set; determine a first image subset belonging to the same moving object and the remaining image subsets excluding the first image subset from the image set; determine the object features of the same moving object based on the first image subset; search for a supplementary image from the remaining image subsets based on the object features; and obtain the trajectory of the same moving object based on the first image subset and the supplementary image.

[0125] Optionally, the processor may further execute program code for the following steps: searching for a supplementary image from the remaining image subset according to object features, including: determining a range to be searched; and searching for a supplementary image from the remaining image subset based on the range according to the object features.

[0126] Optionally, the processor may also execute the program code of the following steps: determining the range to be searched, including: sorting the images in the first image subset according to the shooting time to obtain an initial trajectory, wherein the points in the initial trajectory correspond to the images in the first image subset; dividing the initial trajectory to obtain a plurality of trajectory segments, respectively determining the spatiotemporal search range corresponding to each trajectory segment, and using the spatiotemporal search range as the range to be searched.

[0127] Optionally, the processor may also execute the program code of the following steps: determining the spatiotemporal search range corresponding to each trajectory segment respectively, including: determining the spatial search range of the trajectory segment based on the position corresponding to the starting point and the position corresponding to the ending point of the trajectory segment; determining the temporal search range of the trajectory segment based on the time interval between the starting point and the ending point of the trajectory segment; wherein the spatiotemporal search range includes a spatial search range and a temporal search range.

[0128] Optionally, the processor may also execute program code for the following steps: determining object features of the same moving object based on the first image subset, including: using a re-identification model to identify the images included in the first image subset to obtain features of the images in the first image subset; averaging the features of the images in the first image subset to obtain average features, and using the average features as object features of the same moving object.

[0129] Optionally, the processor may also execute the program code of the following steps: searching for a supplementary image from the remaining image subset based on the object features, including: determining a candidate image from the remaining image subset based on the object features; verifying the candidate image using a verification model to obtain a verification result; and determining the candidate image as a supplementary image when the verification result is that the candidate images belong to the same moving object.

[0130] Optionally, the processor may also execute program code for the following steps: determining candidate images from the remaining image subset based on object features, including: obtaining a first eigenvector of the image in the remaining image subset, and a second eigenvector of the object features; determining the distance between the first eigenvector and the second eigenvector; sorting the distances corresponding to multiple first eigenvectors, determining a predetermined number of first eigenvectors with the smallest distances, and determining the images corresponding to the predetermined number of first eigenvectors as candidate images.

[0131] Optionally, the above-mentioned processor can also execute the program code of the following steps: before using the verification model to verify the candidate image and obtaining the verification result, it also includes: obtaining a training sample set, wherein the training sample set includes a positive sample set and a negative sample set, the positive sample set includes samples with the following characteristics: the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are the same moving object, and the negative sample set includes samples with the following characteristics: the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are not the same moving object; training the training sample set to obtain a verification model.

[0132] Optionally, the mobile object includes at least one of the following: a vehicle, a robot, a pet, and an airplane.

[0133] The processor can call information and applications stored in the memory through a transmission device to perform the following steps: receiving a trajectory generation instruction on an interactive interface; receiving a first image in response to the trajectory generation instruction, wherein the first image corresponds to a predetermined moving object; displaying a first trajectory corresponding to the predetermined moving object on the interactive interface, wherein the first trajectory is generated based on a first image subset, and the first image subset is a set of images in an image set that belong to the same predetermined moving object; displaying a supplementary image on the interactive interface, wherein the supplementary image is an image searched from a remaining image subset excluding the first image subset based on object features of the predetermined moving object, and the object features are determined based on the first image subset; and displaying a synthesized trajectory on the interactive interface, wherein the synthesized trajectory is synthesized based on the first trajectory and the supplementary image.

[0134] The processor can call information and applications stored in the memory through a transmission device to perform the following steps: receiving a first instruction requesting a trajectory of the smart car through a control panel of the smart car; in response to the first instruction, displaying a first trajectory on a display panel of the smart car, wherein the first trajectory is generated based on a first image subset, and the first image subset is a set of images in an image set that belong to the same smart car; receiving a second instruction requesting a trajectory of the smart car through the control panel of the smart car; in response to the second instruction, displaying a second trajectory of the same smart car on the display panel of the smart car, wherein the second trajectory is generated based on the first image subset and a supplementary image, and the supplementary image is searched from a remaining image subset based on smart car features, the smart car features are determined based on the first image subset, and the remaining image subset is a set of images in the image set excluding the first image subset.

[0135] It can be understood by those skilled in the art that Figure 9 The structure shown is for illustration only. For example, the computer device may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 9 It does not limit the structure of the above electronic device. Figure 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 9 Different configurations shown.

[0136] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0137] Example 6

[0138] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided. When the instructions in the computer-readable storage medium are executed by a processor of a terminal, the terminal is enabled to perform any of the above-mentioned image processing methods. Alternatively, the computer-readable storage medium may be a non-transitory computer-readable storage medium, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0139] Optionally, in this embodiment, the computer-readable storage medium may be used to store program codes executed by the image processing method provided in the above embodiment.

[0140] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0141] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring an image set; determining a first image subset belonging to the same moving object and the remaining image subsets excluding the first image subset from the image set; determining object features of the same moving object based on the first image subset; searching for a supplementary image from the remaining image subsets based on the object features; and obtaining a trajectory of the same moving object based on the first image subset and the supplementary image.

[0142] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: searching for a supplementary image from the remaining image subset based on object characteristics, including: determining a range to be searched; searching for a supplementary image from the remaining image subset based on the range based on the object characteristics.

[0143] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: determining a range to be searched, including: sorting the images in the first image subset according to shooting time to obtain an initial trajectory, wherein the points in the initial trajectory correspond to the images in the first image subset; dividing the initial trajectory to obtain multiple trajectory segments, respectively determining the spatiotemporal search range corresponding to each trajectory segment, and using the spatiotemporal search range as the range to be searched.

[0144] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: respectively determining the spatiotemporal search range corresponding to each trajectory segment, including: determining the spatial search range of the trajectory segment based on the position corresponding to the starting point and the position corresponding to the ending point of the trajectory segment; determining the temporal search range of the trajectory segment based on the time interval between the starting point and the ending point of the trajectory segment; wherein the spatiotemporal search range includes a spatial search range and a temporal search range.

[0145] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining object features of the same moving object based on the first image subset, including: using a re-identification model to identify the images included in the first image subset to obtain features of the images in the first image subset; averaging the features of the images in the first image subset to obtain average features, and using the average features as the object features of the same moving object.

[0146] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: searching for a supplementary image from the remaining image subset based on object features, including: determining a candidate image from the remaining image subset based on the object features; verifying the candidate image using a verification model to obtain a verification result; when the verification result is that the candidate images belong to the same moving object, determining the candidate image as a supplementary image.

[0147] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: determining candidate images from the remaining image subset based on object features, including: obtaining a first eigenvector of the image in the remaining image subset, and a second eigenvector of the object feature; determining the distance between the first eigenvector and the second eigenvector; sorting the distances corresponding to multiple first eigenvectors, determining a predetermined number of first eigenvectors with the smallest distances, and determining the images corresponding to the predetermined number of first eigenvectors as candidate images.

[0148] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: before adopting the verification model to verify the candidate image and obtaining the verification result, it also includes: obtaining a training sample set, wherein the training sample set includes a positive sample set and a negative sample set, the positive sample set includes samples with the following characteristics: the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are the same moving object, and the negative sample set includes samples with the following characteristics: the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are not the same moving object; training the training sample set to obtain a verification model.

[0149] Optionally, the mobile object includes at least one of the following: a vehicle, a robot, a pet, and an airplane.

[0150] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: receiving a trajectory generation instruction on an interactive interface; receiving a first image in response to the trajectory generation instruction, wherein the first image corresponds to a predetermined moving object; displaying a first trajectory corresponding to the predetermined moving object on the interactive interface, wherein the first trajectory is generated based on a first image subset, and the first image subset is a set of images in an image set that belong to the same predetermined moving object; displaying a supplementary image on the interactive interface, wherein the supplementary image is an image searched from a remaining image subset excluding the first image subset based on object features of the predetermined moving object, and the object features are determined based on the first image subset; and displaying a synthesized trajectory on the interactive interface, wherein the synthesized trajectory is obtained by synthesizing the first trajectory and the supplementary image.

[0151] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: receiving a first instruction requesting a trajectory of a smart car through a control panel of the smart car; displaying a first trajectory on a display panel of the smart car in response to the first instruction, wherein the first trajectory is generated based on a first image subset, and the first image subset is a set of images in an image set that belong to the same smart car; receiving a second instruction requesting a trajectory of a smart car through the control panel of the smart car; displaying a second trajectory of the same smart car on the display panel of the smart car in response to the second instruction, wherein the second trajectory is generated based on the first image subset and a supplementary image, and the supplementary image is searched from a remaining image subset based on smart car features, the smart car features are determined based on the first image subset, and the remaining image subset is a set of images in the image set excluding the first image subset.

[0152] In an exemplary embodiment, a computer program product is further provided. When a computer program in the computer program product is executed by a processor of an electronic device, the electronic device is enabled to perform any one of the above-mentioned image processing methods.

[0153] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0154] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0155] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0156] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0157] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0159] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: include: Get the image collection; Determining, from the image set, a first image subset belonging to the same moving object and remaining image subsets excluding the first image subset; determining object features of the same moving object based on the first image subset and generating an initial trajectory; Searching for a supplementary image from the remaining image subset based on the object feature, wherein the supplementary image is an image searched from the remaining image subset based on the object feature and a range to be searched, the range to be searched being a spatiotemporal search range corresponding to a plurality of trajectory segments, the plurality of trajectory segments being obtained by dividing the initial trajectory, and the dividing of the initial trajectory being divided according to point positions; A trajectory of the same moving object is obtained based on the initial trajectory and the supplementary image.

2. The method according to claim 1, characterized in that Searching for a supplementary image from the remaining image subset according to the object feature comprises: Determine the scope to be searched; The supplementary image is searched out from the remaining image subset based on the range according to the object feature.

3. The method according to claim 2, characterized in that Determine the scope of your search, including: sorting the images in the first image subset according to shooting time to obtain the initial trajectory, wherein the points in the initial trajectory correspond to the images in the first image subset; The initial trajectory is divided into multiple trajectory segments, and a spatiotemporal search range corresponding to each trajectory segment is determined respectively, and the spatiotemporal search range is used as the range to be searched.

4. The method according to claim 3, characterized in that Determine the spatiotemporal search range for each trajectory segment, including: Determining a spatial search range of the trajectory segment based on a position corresponding to a starting point and a position corresponding to an ending point of the trajectory segment; Determining a time search range of the trajectory segment based on a time interval between a starting point and an ending point of the trajectory segment; The spatiotemporal search range includes the spatial search range and the temporal search range.

5. The method according to claim 1, wherein Determining object features of the same moving object based on the first image subset includes: Using a re-recognition model to recognize images included in the first image subset, and obtain features of the images in the first image subset; The features of the images in the first image subset are averaged to obtain an average feature, and the average feature is used as the object feature of the same moving object.

6. The method according to claim 1, characterized in that Searching for a supplementary image from the remaining image subset according to the object feature comprises: determining a candidate image from the remaining image subset according to the object feature; Using a verification model to verify the candidate image to obtain a verification result; If the verification result is that the candidate images belong to the same moving object, the candidate image is determined to be the supplementary image.

7. The method according to claim 6, characterized in that Determining a candidate image from the remaining image subset according to the object feature includes: Obtaining a first feature vector of an image in the remaining image subset and a second feature vector of the object feature; determining a distance between the first eigenvector and the second eigenvector; Distances corresponding to a plurality of first eigenvectors are sorted, a predetermined number of first eigenvectors having the smallest distances are determined, and images corresponding to the predetermined number of first eigenvectors are determined as the candidate images.

8. The method according to claim 6, characterized in that Before using the verification model to verify the candidate image and obtaining the verification result, the method further includes: Obtaining a training sample set, wherein the training sample set includes a positive sample set and a negative sample set, the positive sample set includes samples having the following characteristics: the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are the same moving object, and the negative sample set includes samples having the following characteristics: the moving object corresponding to the image to be verified and the moving object corresponding to the trajectory are not the same moving object; The training sample set is trained to obtain the verification model.

9. The method according to any one of claims 1 to 8, characterized in that The mobile object includes at least one of the following: a vehicle, a robot, a pet, and an airplane.

10. An image processing method, characterized in that: include: Receiving trajectory generation instructions on the interactive interface; In response to the trajectory generation instruction, receiving a first image, wherein the first image corresponds to a predetermined moving object; Displaying, on the interactive interface, a first trajectory corresponding to the predetermined moving object, wherein the first trajectory is generated based on a first image subset, wherein generating based on the first image subset is to sort images in the first image subset based on shooting time, and points in the first trajectory correspond to images in the first image subset, wherein the first image subset is a set of images in an image set belonging to the same predetermined moving object; Displaying a supplementary image on the interactive interface, wherein the supplementary image is an image searched from a remaining image subset excluding the first image subset based on an object feature of the predetermined moving object and a range to be searched, the object feature being determined based on the first image subset, the range to be searched being a spatiotemporal search range corresponding to a plurality of trajectory segments, the plurality of trajectory segments being obtained by dividing the first trajectory, and the dividing of the first trajectory being divided based on point positions; A composite trajectory is displayed on the interactive interface, wherein the composite trajectory is synthesized according to the first trajectory and the supplementary image.

11. An image processing method for a smart car, characterized in that: include: receiving, through a control panel of the smart car, a first instruction requesting a trajectory of the smart car; In response to the first instruction, displaying a first trajectory on the display panel of the smart car, wherein the first trajectory is generated based on a first image subset, wherein generating based on the first image subset comprises sorting images in the first image subset based on shooting time, and points in the first trajectory correspond to images in the first image subset, wherein the first image subset is a set of images in an image set belonging to the same smart car; receiving, through the control panel of the smart car, a second instruction requesting a trajectory of the smart car; In response to the second instruction, a second trajectory of the same smart car is displayed on the display panel of the smart car, wherein the second trajectory is generated based on the first image subset and a supplementary image, the supplementary image is searched from the remaining image subset based on the smart car characteristics and the range to be searched, the smart car characteristics are determined based on the first image subset, the remaining image subset is the set of images in the image set excluding the first image subset, the range to be searched is the spatiotemporal search range corresponding to multiple trajectory segments, the multiple trajectory segments are obtained by dividing the first trajectory, and the division of the first trajectory is dividing the first trajectory based on point positions.

12. An image processing device, characterized in that: include: A first acquisition module is used to acquire an image set; a first determining module, configured to determine, from the image set, a first image subset belonging to the same moving object and remaining image subsets excluding the first image subset; a second determining module, configured to determine object features of the same moving object based on the first image subset and generate an initial trajectory; a search module, configured to search for a supplementary image from the remaining image subset based on the object features, wherein the supplementary image is an image searched from the remaining image subset based on the object features and a range to be searched, the range to be searched being a spatiotemporal search range corresponding to a plurality of trajectory segments, the plurality of trajectory segments being obtained by dividing the initial trajectory, and the division of the initial trajectory being divided according to point positions; A processing module is configured to obtain a trajectory of the same moving object based on the initial trajectory and the supplementary image.

13. A computer device, characterized in that: include: memory and processor, The memory stores a computer program; The processor is configured to execute a computer program stored in the memory, and when the computer program is executed, the processor is enabled to execute the image processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method according to any one of claims 1 to 11.

Images