An intelligent underwater robot target detection system

Abstract

This invention discloses an intelligent underwater robot target detection system, relating to the field of image processing technology. Its key technical features include: a video analysis module tracking time-varying visual components migrating from the target's background to the target area, retaining the pre-entry propagation state and local features of the target entity upon initial entry into the target area, and forming pending visual evidence; a motion analysis module determining the geometric correspondence state and positional order; a source determination module combining the propagation position range, subsequent background observations, and the target's relative background displacement to determine the source of illumination propagation or the source of the entity, and marking the pending visual evidence; and an identity management module calling the confirmed target entity visual evidence and entity source visual evidence to perform target identity preservation and recovery. This invention can reduce the interference of shallow water caustics on target identity association and improve the reliability of continuous target detection.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an intelligent underwater robot target detection system. Background Technology

[0002] Underwater robots typically acquire underwater images or videos continuously using onboard cameras. They then utilize computer vision technologies such as image target detection, video target tracking, visual feature matching, and target identity association to identify and continuously track underwater equipment, operational targets, or other objects to be detected. The aforementioned processing mainly involves analyzing the target regions, local visual features, and their temporal continuity in the image or video data.

[0003] Due to water refraction, scattering, and surface ripples, incident light in shallow water environments can form caustic bright fringes on the background and target surfaces that migrate over time. In continuous video, such caustics may manifest as time-varying visual structures with obvious local gradients and continuous spatial displacements. For example, when an underwater robot continuously inspects a piece of underwater equipment, a caustic bright fringe that was originally located in the background area outside the equipment may gradually move to the surface of the equipment with the surface ripples and cover a local area of ​​the equipment in several consecutive video frames. When the equipment itself undergoes slight movement or the camera's observation position changes, the caustic bright fringe may exhibit a motion relationship and high visual similarity to the target area in a short period of time.

[0004] Existing underwater visual processing technologies can improve target detection and tracking performance under complex lighting conditions through image enhancement, illumination correction, caustic region recognition, optical flow analysis, and target re-identification. During continuous target tracking, the technology can also determine whether to update the target template based on the current detection confidence, target appearance similarity, or tracking stability. Alternatively, it can re-identify candidate targets by combining historical motion information after a target is temporarily lost. These methods can meet the usage requirements of most underwater visual inspection scenarios.

[0005] However, in the specific process of the caustic moving from the background to the surface of the real target, the current state of the visual structure at the time of its formation may not be sufficient to reflect its subsequent physical attribution. The caustic may have a high target response while it is on the target surface and form a temporarily consistent motion appearance with the local area of ​​the target. As the caustic continues to propagate, the visual structure may leave from the other side of the target and re-enter the background. If the target identity association is mainly based on the appearance similarity or motion continuity within the current time period, the visual information formed by the visual structure during the period of covering the target may be included in the target identity association first. Even if it has left the real target, the existing historical association may still continue to affect the target recapture or subsequent identity template update.

[0006] Therefore, in the continuous video target detection process of underwater robots, for time-varying visual structures that propagate from the background to the target surface and whose final entity affiliation cannot be determined at the generation stage, there is still room for improvement in judging the target identity based solely on the current frame or current short-term features. In particular, how to combine the propagation state of the visual structure before it enters the target, the correspondence between the structure and the target entity during the period of target coverage, and its subsequent propagation results to conduct a unified analysis of the complete time history of the same visual structure, and thereby determine whether the visual evidence formed during the target coverage period is suitable as a basis for subsequent target identity association, remains a technical problem that needs further research in the process of underwater video target detection and continuous recognition. Summary of the Invention

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] An intelligent underwater robot target detection system includes: a video analysis module, a motion analysis module, a source determination module, and an identity management module;

[0009] The video analysis module acquires the target area in the underwater robot video, tracks the time-varying visual components that migrate from the background outside the target to the target area, retains the local features of the target entity in the pre-entry propagation state and the entry position when it first enters the target area, and associates the observations after entry as pending visual evidence.

[0010] The motion analysis module determines the geometrically corresponding state based on the local features of the target entity, and determines the propagation direction and the order in which different positions within the target area are passed based on the pre-propagation state.

[0011] The source determination module acquires subsequent observations of the background outside the target after crossing the target area along the propagation direction. When the propagation state, the sequence of observations, and subsequent observations form a continuous propagation relationship and the time-varying visual component is separated from the corresponding position of the local feature of the target entity, the source of the light propagation is determined. When the propagation position range determined based on the entry into the propagation state is separated from the corresponding position of the local feature of the target entity, the target is displaced relative to the background, and the time-varying visual component still maintains a geometric correspondence, and the continuous propagation relationship is excluded based on the observation of the background outside the target, the entity source is determined, and pending visual evidence is marked according to the determined source.

[0012] The identity management module calls upon visual evidence of the target entity that has been confirmed before the time-varying visual component first enters the target area, as well as visual evidence of the entity's source, to perform target identity preservation and target identity restoration.

[0013] This invention provides an intelligent underwater robot target detection system, which has the following beneficial effects:

[0014] When a time-varying visual component first enters the target area from the background outside the target, this invention retains both its pre-entry propagation state and the existing local features of the target entity at the entry position, and uniformly registers the continuous observations after entry as pending visual evidence. Compared with directly judging identity based solely on the appearance similarity of the current frame, detection confidence, or short-term motion continuity, this processing can postpone identity access when caustic bright fringes have just covered the real target and the physical source cannot yet be determined, thus avoiding premature writing of transient lighting structures into the target identity association.

[0015] This invention further establishes independent propagation relationships and entity local geometric correspondences simultaneously for the same pending visual evidence, and completes source closure by combining the order in which time-varying visual components pass through different positions within the target area and subsequent observations in the background on the other side of the target. When entering the pre-propagation state, the migration process within the target area, and the observation of the departing side background form a continuous propagation relationship, and the time-varying visual component has already detached from the corresponding position of the target entity's local features, it is determined to be the source of illumination propagation. When the independent propagation position has separated from the local position of the target entity, but the time-varying visual component still maintains local geometric correspondence during the target's relative background displacement, it is determined to be the source of the entity. Thus, it is possible to distinguish caustic propagation from real target entity information based on the complete time history of the same visual structure, reducing source misjudgment caused by continuous caustic migration.

[0016] After the source is determined, the present invention back-links the source result to the historical pending visual evidence corresponding to the same first entry event, and only uses the visual evidence of the target entity that has been confirmed before the first entry and the visual evidence that has been determined as the source of the entity for subsequent target identity preservation and target identity recovery; therefore, it can prevent the transient brightness, local texture and motion information formed by caustic bright fringes during the coverage of the target from continuing to inherit the original target identity, reduce its interference with target recapture and identity template update, and improve the reliability of underwater robot target detection, continuous tracking and identity recovery in shallow water complex lighting environment. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating the cross-target propagation and source closure of the time-varying visual components of the present invention;

[0018] Figure 2 This is a flowchart of the source determination module for this invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1 and Figure 2 This embodiment provides an intelligent underwater robot target detection system, which consists of a video analysis module, a motion analysis module, a source determination module, and an identity management module.

[0021] The system uses the event of the first entry of the time-varying visual component into the target area as the processing unit, and performs continuous processing on the same time-varying visual component in the continuous video of the underwater robot. The video analysis module first obtains the pre-entry propagation state of the time-varying visual component in the background outside the target, and determines the local features of the target entity that already exists at the entry position when it first enters the target area. The continuous observations after entry are registered as pending visual evidence.

[0022] The motion analysis module simultaneously calculates the geometric correspondence of the same observation in the visual evidence to be judged with respect to the local features of the target entity, and records the order in which the time-varying visual component passes through different positions in the target area along the forward propagation direction.

[0023] The source determination module reads background observations outside the target area after the target area is crossed along the propagation direction. It determines the source of light propagation based on the propagation state before entry, the order of different positions within the target area, and the continuity between observations on the departure side. After the independent propagation position has separated from the local position of the target entity, it determines the entity source based on the local geometric correspondence that is maintained during the target's relative background displacement.

[0024] Based on the source determination result, the identity management module extracts visual evidence for target identity matching from the entity source observation, and uses the visual evidence to perform target identity preservation and target identity restoration.

[0025] like Figure 1 As shown, the time-varying visual component first forms a continuous migration in the background outside the target, and retains its pre-entry propagation state when it first enters the target area, while determining the local features of the target entity that already exist at the entry position. The continuous observations after the time-varying visual component enters the target area are preserved as pending visual evidence. In subsequent observations, if the time-varying visual component continuously passes through the target area along the pre-entry propagation direction and re-enters the background outside the target from the far side of propagation, while the local features of the target entity remain at the corresponding position of the target entity, it indicates that the visual component has a continuous propagation relationship independent of the target entity, and the source of light propagation can be determined accordingly. If the independent propagation position obtained from the pre-entry propagation state has separated from the local position of the target entity, and the time-varying visual component still maintains a geometric correspondence with the local position of the target entity after the target undergoes a relative background displacement, the source of the entity can be determined by combining the observations from the far side of propagation, thus maintaining a continuous correlation between the pre-entry state, the target coverage process, and the subsequent source results of the same time-varying visual component.

[0026] The target region is the spatial support region of the current target in the video frame, and the background outside the target is the image region outside the target region that can provide a stable background structure.

[0027] Time-varying visual components are local visual structures that have cross-frame spatial migration continuity in continuous video, and their brightness, area, and contour can change with the acquisition time.

[0028] Entering the pre-propagation state refers to the history of propagation direction, position along the propagation direction, lateral deviation, and scale changes that have already formed in the background outside the target before the time-varying visual component first enters the target area.

[0029] Local features of a target entity refer to entity structures that existed near the entry position before the arrival of the time-varying visual component and can maintain cross-frame geometric correspondence with neighboring structures on the target.

[0030] Pending visual evidence refers to continuous post-entry observations indexed by the same first entry event, which maintain a fixed association with the pre-entry propagation state and local features of the target entity until the source is determined.

[0031] The underwater robot continuously collects video of the water area where the target is located through a camera and establishes a frame sequence according to the original acquisition timestamp; target detection and target segmentation use existing image detection methods to obtain the target region.

[0032] For the background outside the target, extract ORB key points and calculate ORB descriptors in the local background around the target outside the target area and the coverage of the current pending visual evidence; perform bidirectional nearest neighbor Hamming distance matching on the ORB descriptors in adjacent video frames, and retain only the matching point pairs that are the nearest neighbors of each other in consecutive frames.

[0033] When there are at least 3 effective matching point pairs and the corresponding background points are not collinear in the image plane, the minimum median method is used to estimate the two-dimensional affine transformation, and the two-dimensional affine transformation is used as the background viewpoint compensation mapping of adjacent video frames to convert the current frame position to the unified background coordinates of the previous frame; the effective matching point pairs are preferably 6 to 12, and even more preferably 8.

[0034] The effectiveness of background viewpoint compensation mapping is determined based on the reprojection residuals in the stable historical background. The reprojection residuals of the effective background matching points in the stable historical frames after two-dimensional affine transformation are sorted by value, and the corresponding values ​​of the 85% to 95% empirical quantiles are selected. The 90% empirical quantile is further optimized and compared with the minimum quantization step size of the ORB keypoint coordinates. The larger value is taken as the background mapping positioning tolerance. When the reprojection residual of the effective matching points in the current frame exceeds this tolerance, the background viewpoint compensation result of the frame is recorded as invalid. Two-dimensional affine transformation is obtained using the local background around the target, so that the compensation range is limited to the local field of view used for subsequent propagation position comparison. When there is a significant depth parallax in the local background and the reprojection residual cannot meet the tolerance, the current frame does not participate in the propagation state calculation under the unified background coordinates.

[0035] For the target region, the solution point pairs for the target local geometry mapping are taken from neighboring target features that have been stable and located around the local features of the target entity before the time-varying visual component first enters. The target entity local features themselves and the current time-varying visual component do not participate in the parameter solution of the target local geometry mapping. When a certain neighboring target feature is covered by the time-varying visual component, the current target local geometry mapping is obtained only when the effective corresponding point pairs provided by the remaining still visible neighboring target features reach the minimum number of solutions for the selected geometric transformation, and the historical reference position of the target entity local features is converted to the current frame through this mapping. When there are insufficient effective corresponding point pairs, the current acquisition time is recorded as the invalid time of the target local geometry mapping.

[0036] The parameters of the local geometric mapping of the target are obtained independently from the features of neighboring targets, and the time-varying visual components in the pending visual evidence do not participate in the fitting of the mapping parameters.

[0037] In a preferred embodiment, the target local geometric mapping employs a similarity transformation, which is determined by at least two sets of non-overlapping point pairs, preferably three to five sets of effective corresponding point pairs, and more preferably four sets. The corresponding point pairs are preferentially provided by the neighboring target structures around the local features of the target entity. In cases where there is obvious shearing on the local surface and more stable key points can be obtained, the target local geometric mapping employs an affine transformation, preferably four to seven sets of effective corresponding point pairs, and more preferably six sets.

[0038] The target local mapping and positioning tolerance is determined using the leave-one-out projection residuals from stable target observations prior to the first entry. At each valid historical moment before the first entry, a leave-one-out calculation is performed sequentially for each set of stable corresponding points at that time: the corresponding points left out this time do not participate in the solution of mapping parameters, and the target local geometric mapping is obtained from the other valid corresponding points. Then, the left-out points are mapped to the reference coordinates at the first entry moment, and the Euclidean distance between the mapped position and the reference position of the point is calculated as one leave-one-out projection residual. Only the leave-one-out results that still satisfy the minimum number of solutions for the selected geometric transformation after leaving out one set of corresponding points are retained. The leave-one-out projection residuals obtained from each valid historical moment are merged and sorted by value. The corresponding values ​​of 85% to 95% empirical quantiles are taken, and the 90% empirical quantile is further optimized. The value is compared with the minimum quantization step size of the coordinates output by the local feature positioning algorithm, and the larger value is taken as the target local mapping and positioning tolerance.

[0039] By leaving one out, the local mapping and positioning tolerance of the target is determined by the verification points that did not participate in this mapping fitting. In this embodiment, the empirical quantiles are all calculated using the nearest rank method: the effective samples are sorted in ascending order of value, and the index obtained by rounding up the product of the total number of samples and the quantile ratio is used as the position of the value, and the sample value corresponding to the index is read. This calculation does not require the samples to follow a normal distribution.

[0040] The stable reference position of the local features of the target entity is determined by the median value of the coordinates of the position obtained by converting the local features of the target entity to the reference coordinates at the time of the first entry in the effective observations before the first entry.

[0041] The historical geometric fluctuations of the target entity's local features and the positioning errors of the target's local geometric mapping are recorded separately. The subsequent judgment of the geometric correspondence state uses the historical position residuals of the target entity's local features to form the normal geometric range.

[0042] The uncertainty range of the target boundary is formed by the target's stable video prior to the first entry event.

[0043] After completing the background viewpoint compensation, the target boundary is divided into multiple boundary segments according to approximately equal arc lengths. Each boundary segment retains a representative normal position residual at each effective time. The representative residuals obtained at different effective times and different boundary segments are then merged into a set of historical target boundary normal residual displacements.

[0044] The representative residuals are sorted in ascending order of value. The corresponding values ​​of 85% to 95% empirical quantiles are selected according to the nearest rank method determined in this embodiment. The 90% empirical quantile is further optimized and compared with the minimum quantization step size of the target boundary output coordinates. The larger value is taken as the boundary uncertainty radius.

[0045] The minimum quantization step size of the target boundary output coordinates is determined by the accuracy of the boundary coordinate representation of the target detection or target segmentation results; when using pixel-level boundaries, one pixel is taken, and when using sub-pixel boundaries, one minimum output step size of the corresponding sub-pixel coordinates is taken.

[0046] The boundary uncertainty radius is determined by the corresponding camera, imaging distance, target category, and target boundary positioning resolution, and does not use fixed pixel values ​​that are independent of imaging conditions.

[0047] When the 90% empirical quantile is used as a further optimized value, the boundary uncertainty range can cover the main normal fluctuations in the stable history and reduce the impact of tail-end abnormal residuals on the boundary range.

[0048] The same boundary uncertainty radius is used for subsequent first entry into and exit from the target area and for location recording intervals.

[0049] When the video analysis module forms time-varying region candidates in the background outside the target, it first uses background viewpoint compensation mapping to convert adjacent video frames to the same background coordinates and converts the color image into a brightness map. It calculates the absolute value of the brightness difference between corresponding pixels in adjacent brightness maps and uses the Otsu method to automatically determine the binary segmentation threshold based on the grayscale histogram of the current background difference map outside the target. Pixels whose difference reaches the segmentation threshold are recorded as changed pixels.

[0050] For the changing pixels, the connected components are marked according to the 8-neighbor connectivity relationship. Each connected component forms a time-varying region candidate, and the pixel set, pixel area and area weighting center of the candidate are recorded. The pixel set is the region support of the candidate in the current video frame. This step only forms the candidate region of the time-varying visual component, and does not distinguish the physical source of light propagation, target surface highlights and suspended object reflections at this stage.

[0051] For the already formed time-varying region candidates, the Lucas-Kanade sparse optical flow of the pyramid is used to track the trackable corner points in the candidate region. The candidate region of the previous frame is shifted according to the median value of the effective corner point optical flow displacement to form the candidate search window of the next frame. Then, within the search window, the newly formed connected components are associated across frames according to the center position after background compensation, the regional overlap relationship and the migration direction.

[0052] When registering time-varying visual components for the first time, it is preferred that the candidate maintains the same migration process for 3 to 5 consecutive valid acquisition times, and 3 valid acquisition times are further preferred. The 3 times form 2 adjacent displacements, which can confirm the continuous migration direction and distinguish single-frame flicker. More than 5 times will increase the confirmation delay near the first entry event.

[0053] After registration is completed, subsequent observations continue to access the time-varying visual component along the same cross-frame association chain.

[0054] When the time-varying visual component reaches the first entry judgment condition, the integrity check is performed on the background history outside the target that has been formed before the first entry; the pre-entry propagation state used in this first entry event is only formed by the effective observations before the first entry, and the observations after the first entry continue to be saved as post-entry observations and do not participate in the supplementary calculation of the pre-entry propagation state of this event.

[0055] For each valid observation, first calculate the pixel area of ​​the time-varying visual component, then calculate the area weighting center using the pixel coordinates within the region; divide the region area by pi and take the square root, then use the radius of the area equivalent circle as the current equivalent scale.

[0056] The lateral drift is normalized using an equivalent scale, so that time-varying visual components of different sizes can be compared using a consistent relative lateral drift scale.

[0057] Before entering the target area for the first time, at least 5 valid acquisition times should be retained, preferably 6 to 10 under normal operating conditions, and even better, 8 times. Observations that are closest to the first entry time and have continuous cross-frame correlation should be given priority.

[0058] Using 5 effective time points, 4 adjacent propagation states and 3 state transition relationships can be formed, which satisfies the basic time sequence calculation of the subsequent 3-component propagation states; using 8 further optimized effective time points, 7 propagation states and 6 state transition relationships can be formed, which retains redundant samples for short-time propagation state evolution calculation.

[0059] Before entering the pre-propagation state, it is also checked whether the history contains identifiable real migrations.

[0060] Specifically, candidate propagation directions are first determined based on the area-weighted center displacement of adjacent valid observations before the first entry. The candidate propagation directions are calculated by minimizing the sum of the unit migration direction and Euclidean distance, which are the same as the propagation directions before entry. Then, the projection length of the center displacement from the first valid observation to the last valid observation on the candidate propagation direction is calculated. This projection length is taken as the net migration distance and compared with twice the boundary uncertainty radius and the median value of the equivalent scale before entry. The larger of the two values ​​is taken as the minimum identifiable migration amount of this event.

[0061] Twice the radius of the boundary uncertainty corresponds to the full width of the uncertainty zone of the target boundary, and the median value of the equivalent scale before entering corresponds to the typical spatial size of the time-varying visual component.

[0062] When the net migration distance reaches this larger value and the candidate propagation direction passes the leave-one stability check of the pre-entry propagation direction, the candidate propagation direction is registered as the pre-entry propagation direction, and the pre-entry propagation state is calculated accordingly; if any condition is not met, the first entry event does not form a pre-entry propagation state that can be used for source determination.

[0063] The initial entry into the target area is determined based on the positional changes of the subject relative to the target boundary using time-varying visual components.

[0064] For each pixel within the time-varying visual component region, calculate its shortest distance to the target boundary. Record the inside of the target region as positive values ​​and the background outside the target as negative values. Then sort all signed distances and take the median value as the current subject position.

[0065] The preferred length of continuous confirmation for the initial entry judgment is 2 to 3 valid acquisition times, and more preferably 2: the median value of at least 2 consecutive valid times before entry is outside the negative boundary uncertainty radius, and then the median value of 2 consecutive valid times is inside the positive boundary uncertainty radius.

[0066] When the second consecutive valid inner observation arrives and continuity is confirmed, the first acquisition time in the consecutive inner observation is retrospectively recorded as the first entry time.

[0067] When the inner condition is met for two consecutive valid moments, the first entry state can be continuously confirmed, and the confirmation delay of the first entry moment can be kept small.

[0068] Table 1. Initial Entry Event and Basic Observation Parameters:

[0069]

[0070] The propagation direction before entry is determined only by valid observations prior to the first entry time.

[0071] First, calculate the area-weighted center displacement between every two adjacent valid observations, remove each value and obtain the unit migration direction by its own length; then, find a unit direction that minimizes the sum of the Euclidean distances between this direction and all unit migration directions, and use this direction as the forward propagation direction.

[0072] To verify whether the direction is dominated by a single anomalous observation, one unit migration direction was removed sequentially and the projections of the remaining directions onto the propagation direction before entry were repeatedly checked; if each projection remained forward, the direction was confirmed to have stable directional significance.

[0073] Once the propagation direction is established before entry, the order of entry within the target area, the far side of propagation, and the subsequent background search area outside the target are all established in the same direction.

[0074] The pre-propagation state consists of four types of quantities arranged in the order of the original acquisition time.

[0075] Starting from the area-weighted center of the first valid observation before entry, the displacement vector is obtained by subtracting the starting point from the current area-weighted center. The projection length of this displacement vector in the propagation direction before entry is taken as the position along the propagation direction, and the projection length in the normal direction of the propagation direction is taken as the lateral deviation, which is saved together with the current equivalent scale.

[0076] The system records the positional changes of time-varying visual components along the propagation direction, their lateral deviation relative to the propagation direction, and scale changes during the pre-propagation state.

[0077] After the initial entry time, new axial position, lateral deviation and equivalent scale are calculated along this fixed propagation direction to ensure that the propagation records before and after entry have the same coordinate reference.

[0078] When the time-varying visual component first enters the target region, the local features of the target entity are determined simultaneously.

[0079] Specifically, ORB key points are detected and ORB descriptors are calculated within the target area obtained before the first entry. Bidirectional nearest neighbor Hamming distance matching is performed on the ORB descriptors in adjacent valid observations to form cross-frame correspondence of key points within the target area. The target local geometric mapping obtained by using the features of neighboring targets is used to convert the positions of each historical key point to the target local coordinates at the time of the first entry.

[0080] Keypoints that can continuously establish cross-frame correspondences in all valid pre-entry observations used in this first entry event, and whose deviations from their own coordinate median values ​​at each historical moment are not greater than the target local mapping positioning tolerance, are identified as stable keypoint candidates. The Euclidean distance between the position of each stable keypoint candidate in the target local coordinates at the first entry moment and the first entry position of the time-varying visual component is calculated, and the stable keypoint candidate with the smallest distance is selected first. If there are candidates with the same distance, the candidate with the smaller median value of the historical mapping position residual is selected. The position, orientation, and ORB descriptor of the selected keypoint are jointly registered as local features of the target entity.

[0081] Around the local features of the target entity, 2 to 4 neighboring target features are selected from the remaining stable keypoint candidates in order of distance from near to far, and 3 are further selected. The selected neighboring target features are arranged to have different spatial orientations in the image plane for subsequent local geometric mapping of the target. The local features of the target entity itself continue to serve as independent verification anchor points and do not participate in the fitting of the mapping parameters.

[0082] When using similarity transformation under normal operating conditions, 3 to 5 sets of effective corresponding point pairs are obtained from the features of the neighboring target, and 4 sets are further selected; when using affine transformation, effective corresponding point pairs are obtained according to the aforementioned number of corresponding point pairs for affine transformation.

[0083] Within the eight preferred effective time points prior to the first entry, the local geometric mapping of the target is first obtained from the features of the neighboring targets at each time point, and then the local features of the target entity that were actually detected and matched at that time are converted to the reference coordinates of the first entry time. The two-dimensional difference between the mapped position and the stable reference position of the local features of the target entity is used as the local position residual of the target entity at that time point.

[0084] The source of the local features of the target entity and the features of neighboring targets is fixed as the target structure that existed before the first entry. After the first entry, only the current corresponding position and the verification residual are updated.

[0085] The video analysis module then establishes pending visual evidence using the same time-varying visual component and the same first entry time.

[0086] The event log stores the first entry event index, time-varying visual component index, entry pre-propagation state index, target entity local feature and neighboring target feature index, and adds the acquisition time, time-varying visual component region support, background compensation position, target local position and source state field frame by frame starting from the first entry time; the source state field is registered as pending state when it is first written.

[0087] Each post-entry observation is simultaneously associated with the pre-entry propagation state and the local features of the target entity. The pre-entry propagation state records the independent propagation history of the visual structure before it enters the target, while the local features of the target entity record the original local structure of the target entity at the entry position.

[0088] The motion analysis module and the source determination module read the corresponding event records according to the first entry event index, so that the subsequent source conclusions can accurately correspond to the historical observations of this first entry event.

[0089] The motion analysis module synchronously calculates the entity local relations and independent propagation relations at each valid moment in the visual evidence to be dealt with.

[0090] The calculation of local entity relationships uses the features of neighboring targets as the input for solving the local geometric mapping of the target, and converts the time-varying visual components and the local features of the target entity as the objects to be verified to the target local coordinates at the first entry time.

[0091] The historical normal geometric range is formed only by the historical position residuals of the local features of the target entity itself. The two-dimensional residuals, the corresponding reverse vectors and the zero displacement points are taken as the centrally symmetric convex hulls. When the resulting convex hull has a two-dimensional area, it is directly used as the normal geometric range.

[0092] When the historical position residuals are approximately collinear or concentrated near zero, causing the convex hull to degenerate, the minimum and maximum values ​​of the lateral and longitudinal residuals are obtained respectively, and the local mapping positioning tolerance of the target is extended outward in both coordinate directions to form an axis-aligned rectangle as the normal geometric range.

[0093] At the first entry moment, the initial local offset of the area-weighted center of the time-varying visual component relative to the local features of the target entity is recorded. For each subsequent valid observation, the local geometric mapping of the target is first obtained from the neighboring target features, and the area-weighted center of the time-varying visual component is converted to the reference coordinates at the first entry moment. When the local features of the target entity are currently visible, their actual observation position is also converted. When they are currently covered, their stable reference position is used. The current local offset is obtained by subtracting the current reference position of the local features of the target entity from the current mapped position of the time-varying visual component. Then, the current relative offset change is obtained by subtracting the initial local offset at the first entry moment from the current local offset.

[0094] Then, the smallest non-negative scaling factor that can make the normal geometric range include the current relative offset change after scaling is obtained is used as the geometric corresponding scale; when the current relative offset change is zero, the geometric corresponding scale is 0.

[0095] During engineering calculations, the normal geometric range without scaling is first checked at a scale of 1. If the current relative offset change is within this range, a binary search is performed between 0 and 1. If it is outside the range, the scale is increased by a factor of 2 starting from 1 until the first upper bound that can contain the current relative offset change is obtained. Then, a binary search is performed between the previous scale and this upper bound.

[0096] The search termination condition is that the change in the current boundary of the normal geometric range is not greater than the local mapping and positioning tolerance of the target; when the obtained minimum ratio is not greater than 1, the registered geometric corresponding state is maintained, and when it is greater than 1, the registered time-varying visual component is separated from the corresponding position of the local feature of the target entity.

[0097] Table 2, Target Local Geometric Mapping Parameters:

[0098]

[0099] The normal geometric range is formed using only stable observations saved prior to the first entry event when the event is confirmed; observations acquired after the first entry are not written back into the normal geometric history of that event.

[0100] The motion analysis module simultaneously saves the geometrically corresponding scale and the corresponding acquisition time for each valid moment in the visual evidence to be dealt with, so as to make hourly comparisons with the propagation state at the same moment.

[0101] The geometric correspondence scale only indicates whether the relative position of the time-varying visual component with respect to the local features of the target entity is still within the normal geometric fluctuation range previously confirmed when it first entered the target entity.

[0102] When the local features of the target entity are temporarily invisible, the relative offset calculation is continued using the local geometric mapping of the target obtained from the nearby target features and the stable reference position of the local features of the target entity; after the local features of the target entity become visible again, the subsequent verification records are updated with their actual mapping positions.

[0103] The normal geometric range and initial local offset are formed from stable observations saved prior to the first entry event when the first entry event is confirmed. Observations after the first entry event are not written back as historical references for this event.

[0104] In this computational relationship, the target local geometric mapping is used to unify the target local coordinates, the target entity local features serve as the entity position reference, and the time-varying visual components serve as the verification object for the geometric correspondence state.

[0105] Independent propagation relationships and entity local relationships are formed synchronously using observations at the same time.

[0106] The motion analysis module records the directional position of the time-varying visual components along the forward propagation direction in the uniform background coordinates after background viewpoint compensation.

[0107] The area-weighted center at the moment of first entry is used as the first location recording point. Subsequently, when the cumulative projected displacement of the current area-weighted center relative to the most recent location recording point along the propagation direction before entry reaches the larger value between twice the boundary uncertainty radius and the equivalent scale at the moment of first entry, and the projection direction remains forward, the next location recording point is registered.

[0108] The time-varying visual components pass through different locations within the target area in the order determined by the actual arrival time of each location recording point.

[0109] The location interval takes into account both the uncertain bandwidth of the target boundary and the scale of the time-varying visual components themselves, so there is no need to pre-divide the target area into a fixed number of equal-width regions.

[0110] For two adjacent valid observations, the propagation state is calculated using three components.

[0111] The first component is obtained by subtracting the position of the previous moment from the position of the next moment along the propagation direction, and then dividing by the actual sampling time interval.

[0112] The second component is obtained by subtracting the lateral deviation of the previous moment from the lateral deviation of the next moment, and then dividing by the product of the actual sampling time interval and the equivalent scale of the previous moment, so as to obtain the lateral drift velocity normalized relative to the scale of the time-varying visual component itself.

[0113] The third component is obtained by first taking the ratio of the equivalent scale at the next time step to the equivalent scale at the previous time step, taking the natural logarithm of this ratio, and then dividing it by the actual sampling time interval to obtain the relative rate of scale change.

[0114] The three components are fixed and stored in the order of directional propagation velocity, normalized lateral drift velocity, and relative scale change velocity, and are associated with the corresponding acquisition time.

[0115] The effective states that maintain forward migration along the pre-entry propagation direction from the propagation states obtained before the first entry are formed into the allowable range of the pre-entry propagation state. First, the value span of the three propagation state components in the pre-entry history is calculated respectively. The component with a value span greater than the numerical calculation precision of the corresponding component is regarded as the variable component, and the convex hull of the historical state is taken in the subspace composed of the variable components. The component with a value span not greater than the numerical calculation precision of the corresponding component is regarded as the approximately constant component. The allowable interval is formed with the historical median value of the component as the center and the corresponding numerical calculation precision. When the subsequent propagation state simultaneously satisfies that the variable component is located in the convex hull and the approximately constant component is located in the allowable interval, it is registered as compatible with the pre-entry propagation state.

[0116] Each propagation state after entering the target area is calculated using the same three terms, and the changing components and approximately constant components are checked simultaneously to see if they meet the allowable range of the propagation state before entering the target area.

[0117] The numerical calculation accuracy of each propagation state component is obtained by finite perturbation calculation with the minimum input quantization step size. For the position along the propagation direction, lateral deviation, equivalent scale, and original timestamp that form a certain propagation state component, positive and negative 1 corresponding minimum input quantization step sizes are applied to the original values ​​respectively, and the propagation state component is recalculated. The maximum value of the absolute difference between each recalculation result and the original result is taken as the numerical calculation accuracy of the component. When uniform propagation or constant scale causes the historical state to be in a low-dimensional subspace, the comparison is made according to the convex hull of the subspace corresponding to the changing component and the allowable interval corresponding to the approximately constant component. Observations after the first entry are not used to supplement the pre-entry history of this event.

[0118] For each valid moment of the same pending visual evidence, the geometrical scale relative to the local features of the target entity and the propagation state compatibility relative to the pre-entry propagation history are recorded simultaneously.

[0119] When the projection distance between the most recent location recording point and the far-side boundary along the direction of propagation before entry is not greater than the currently used location recording interval, the source determination module determines the far-side search area of ​​the background outside the target.

[0120] Specifically, in the unified background coordinates after background viewpoint compensation, each point of the current target boundary is projected onto the propagation direction before entry, and the boundary position corresponding to the maximum projection value is taken as the far-side boundary of propagation; the region outside the target area and whose projection value along the propagation direction before entry is greater than the sum of the maximum projection value and the uncertain radius of the boundary is taken as the background outside the far-side target of propagation.

[0121] Continuous time-varying observations were then extracted from the area.

[0122] The independent propagation location range is also calculated in a unified background coordinate system. The part of the propagation location range that satisfies the same projection conditions is taken as the far side of the propagation. The subsequent observation location and the propagation location range are compared in the same coordinate system accordingly.

[0123] The range of independent propagation locations is established within the permissible range of propagation state evolution.

[0124] The samples used to form the propagation state evolution history are preferentially taken from continuous time-varying observations within the same continuous video segment, which are located outside the background of the target throughout the entire visible life cycle, are in the same direction as the current pre-entry propagation direction, and whose 3-component propagation state falls within the allowable range of the pre-entry propagation state; these observations are only used as samples of propagation state changes and are not pre-assigned with a light propagation source label.

[0125] For every two adjacent propagation states, subtract the corresponding component of the previous propagation state from the current three propagation state components, and then divide by the actual sampling time interval between two valid observations to obtain three state transition rate components.

[0126] At least 4 groups of effective state transition rate samples should be taken, preferably 4 to 8 groups, and even more preferably 6 groups. If there are insufficient complete background samples outside the target, the effective state transition rate formed in the previous propagation state when the current first enters the target should be used to supplement the sample. If there are still less than 4 groups after merging the two types of samples, the independent propagation location range will not be calculated for the visual evidence pending this time, and the source status will remain pending.

[0127] For each component, the median value of the effective samples is taken and combined to form the propagation state transition center. Then, the deviation vector is obtained by subtracting the propagation state transition center from each group of state transition rates. For components whose deviation span is greater than the numerical calculation precision of the corresponding state transition rate, the centrally symmetric convex hull of the deviation vector is taken in the subspace composed of the changing components. For components whose deviation span is not greater than the corresponding numerical calculation precision, an allowable interval is formed with 0 as the center and according to the corresponding numerical calculation precision. The convex hull and the allowable interval together constitute the allowable range of propagation state transition deviation.

[0128] The numerical calculation accuracy of the propagation state transition rate is determined using the same finite perturbation method: positive and negative perturbations are applied to the previous propagation state, the current propagation state, and the two acquisition times according to their respective numerical calculation accuracy or minimum time quantization step size, the state transition rate is recalculated, and the maximum value of the absolute value of the difference between the recalculated result and the original result is taken as the numerical calculation accuracy of the corresponding state transition rate component.

[0129] Starting from the actual propagation state at the end of the pre-propagation state, the equivalent scale at the end, and the spatial position at the end, a joint recursive branch is established. Each joint recursive branch simultaneously stores three propagation state components, one equivalent scale, and a 2D spatial position in a unified background coordinate system, which together serve as the input for the recursion at the next moment.

[0130] For each joint recursive branch of the previous time step, the propagation state transition center and the boundaries of the allowable range of propagation state transition deviation are combined to form the allowable state transition rate. Then, each allowable state transition rate is applied to the joint recursive branch. The three components of the allowable state transition rate are multiplied by the current real sampling time interval and added to the corresponding three components of the propagation state of the previous time step to obtain the candidate propagation state of the next time step.

[0131] When the first component of a candidate propagation state is less than or equal to 0, the candidate does not enter the subsequent forward propagation branch; the remaining candidates update the equivalent scale of the next moment with the third component, multiply the first component by the time interval to obtain the displacement along the propagation direction, multiply the second component by the equivalent scale of the previous moment and the time interval to recover the actual lateral displacement, and then superimpose the two displacements onto the spatial position of the previous moment to obtain the candidate spatial position of the next moment.

[0132] After completing all combinations, the three propagation state components, equivalent scale, and 2D spatial position of each candidate are merged into a 6D joint state. First, repeated joint states are merged according to the numerical calculation accuracy of each dimension. Then, the dimensions with actual value spans of these joint states are determined, and a convex hull is taken in the affine subspace formed by these dimensions. Only the vertices of the joint convex hull are retained as the recursive branches for the next time step. The propagation position range is obtained by projecting the 2D spatial positions of the vertices of these joint convex hulls and then taking a 2D convex hull. Thus, the propagation state, equivalent scale, and spatial position maintain the same branch correspondence during the recursive process.

[0133] When the joint state changes in only one dimension, the joint state of the two endpoints of that dimension is retained; when no dimension changes beyond the accuracy of numerical calculation, only one joint state is retained to continue the recursion.

[0134] The propagation location range obtained by the joint recursive branch projection serves as an independent propagation spatial benchmark in subsequent determinations of the source of light propagation and the source of the entity.

[0135] For each subsequent observation candidate in the background outside the target, the pre-propagation state, the propagation record in the target area and the candidate observation are connected into a complete time series according to the acquisition time, and then the state transition rate is calculated segment by segment.

[0136] For each state transition rate, first subtract the propagation state transition center to obtain the current deviation vector, and then find the smallest non-negative scaling factor that can make the allowable range of propagation state transition deviation include the deviation vector after scaling; when the current deviation vector is zero, the scaling factor is 0.

[0137] The scale search uses the same doubling method as the geometric scale to determine the upper bound and the bisection refinement process.

[0138] After all valid time points have been calculated, the maximum value among the minimum proportions is taken as the propagation continuity scale of the entire time sequence chain.

[0139] When the propagation continuity scale is no greater than 1, each state change in the entire time sequence falls within the historical deviation allowable range of the current scene; the value 1 corresponds to the unscaled deviation allowable range boundary.

[0140] Candidates in the background outside the target are recorded as leaving-side candidates when they meet the following continuity relationships: their first valid position is located in the far side of the propagation position range at the corresponding time; the position record in the pending visual evidence has progressed from the entry position to the far side of propagation over time; after the candidate enters the background outside the target, its projection position along the propagation direction before entry continues to increase.

[0141] The length of the candidate continuous satisfaction of the above relationship is preferably 3 to 5 effective acquisition times, and further preferably 3 times; the 3 times form 2 continuous displacements after leaving the target, which can confirm that the migration on the departure side is still continuing.

[0142] When multiple candidates simultaneously meet the conditions, they are recorded continuously. Only when one candidate reaches the continuous satisfaction length and maintains the propagation continuity scale not greater than 1, that candidate is determined as the subsequent observation of the current pending visual evidence.

[0143] When determining the source of light propagation, the source determination module sequentially connects the pre-propagation state, the order of different locations within the target area, and the subsequent observations of the background outside the target that have been determined, according to the acquisition time. It checks whether the continuity scale of the entire propagation is no greater than 1 and confirms that the subsequent observations are located in the background outside the target on the far side of the propagation.

[0144] Then, the geometrical scale of the time-varying visual components in the local coordinates of the target is read.

[0145] The length of continuous confirmation with a geometric correspondence scale greater than 1 is preferably 2 to 4 effective acquisition times, and more preferably 3. The 3 effective times form 2 continuous separation processes, which can reduce the separation misjudgment caused by local mapping fluctuations of the target in a single frame.

[0146] During this continuous verification period, the left-one-fold projection residual of the neighboring target features used to obtain the local geometric mapping of the target is kept no greater than the local mapping positioning tolerance of the target; when the local features of the target entity are currently visible, the verification residual between its actual mapping position and the stable reference position is kept within the normal geometric range; when the local features of the target entity are temporarily invisible due to being covered by time-varying visual components, the local geometric mapping of the target obtained by the neighboring target features is used to update its reference position.

[0147] When the above relationship is satisfied, the same pending visual evidence will be registered as a source of light propagation.

[0148] The target coverage time boundary of the light propagation source is determined using the same signed distance and propagation direction as the first entry.

[0149] The initial entry time is determined according to the aforementioned continuous inner conditions; after the time-varying visual component passes through the far side of propagation, its signed distance median value re-enters outside the negative boundary uncertainty radius, and at the same time, the projection value of its background-compensated area-weighted center along the propagation direction before entry is greater than the sum of the maximum projection value of the far side boundary and the boundary uncertainty radius.

[0150] The length of continuous confirmation of leaving the target area is preferably 2 to 3 valid acquisition times, and more preferably 2. When the second valid observation that meets the leaving condition arrives and completes the continuity confirmation, the first acquisition time in the continuous leaving observation is back-tracked and registered as the time of leaving the target area.

[0151] Once the source of light propagation is determined, the corresponding continuous observations from the first entry time to the time of leaving the target area are uniformly marked as the source of light propagation, thereby clarifying the time range for source tracing.

[0152] Table 3. Propagation status and subsequent observation parameters:

[0153]

[0154] When determining the source of an entity, the independent propagation location range is not directly input into the target local geometry mapping.

[0155] The source determination module first translates the normal geometric range to the registered position in the target local coordinates at the first entry time, with the first entry position of the time-varying visual component as the center; then, it converts the range to the current image coordinates through the inverse transformation of the current target local geometric mapping, and converts it to the unified background coordinates through the background viewpoint compensation mapping, forming the target entity local corresponding region at the current time.

[0156] Since the independent propagation locations are already located in a unified background coordinate system, the two can be directly compared within the same coordinate system.

[0157] The value of the corresponding area of ​​the target entity is converted to a unified background coordinate according to the local mapping and positioning tolerance of the target entity and then extended outward. When the propagation position range and the corresponding area of ​​the expanded target entity have no intersection for 2 to 3 consecutive effective acquisition times, and 2 times are further selected, the independent propagation position and the local position of the target entity are registered to form an identifiable separation.

[0158] The target displacement relative to the background is determined by the common position change of effective target features in a unified background coordinate system. When the local features of the target entity are currently visible, they are used together with the features of neighboring targets as effective target features. When the local features of the target entity are temporarily invisible, no less than three currently visible neighboring target features are used as effective target features.

[0159] When entering for the first time, it is preferable to configure 2 to 4 neighboring target features in the local feature configuration of the target entity, and more preferably 3; when performing the target displacement calculation relative to the background, at least 3 currently visible valid target features should be retained.

[0160] For the first entry into the previous stable history, the median value of the coordinates of the background compensation position of each target feature is first taken as the stable reference position of the feature; at each historical moment, the displacement vector of each target feature relative to its own stable reference position is calculated, and the median value of the horizontal and vertical components of these displacement vectors is taken to obtain the common displacement vector at that moment.

[0161] The historical common displacement vector, corresponding reverse vector, and zero displacement point are taken as a centrally symmetric convex hull to form the historical fluctuation range of the target relative to the background. At the same time, the common displacement vector at the same moment is subtracted from each effective target feature displacement vector to obtain the local co-motion residual. The local co-motion residual range is formed by the local co-motion residual, the corresponding reverse vector, and the zero displacement point. When any range degenerates due to the historical samples being approximately collinear or concentrated near zero, the historical minimum and maximum values ​​of its horizontal and vertical components are obtained respectively. The range is then expanded outward in both coordinate directions according to the minimum resolvable displacement of the unified background coordinates to form an axis-aligned rectangle. The minimum resolvable displacement of the unified background coordinates is the maximum displacement after the minimum quantization step size of the background feature matching output coordinates is mapped by the background viewpoint compensation.

[0162] The target displacement relative to the background is established when the current common displacement vector is outside the historical fluctuation range of the target relative to the background, and the local co-movement residuals of each target feature are all within the range of local co-movement residuals.

[0163] The target displacement relative to the background is determined by the range of historical common displacement and the range of local co-movement residuals, without setting a separate direction angle threshold or assuming that the historical displacement follows a specific probability distribution.

[0164] After the independent propagation location range has formed a identifiable separation from the local location of the target entity, the source determination module continues to calculate the geometric corresponding scale of the time-varying visual components during the period when the target displacement relative to the background is established.

[0165] When the time-varying visual component is still within the normal geometric range of the local features of the target entity, the geometric corresponding scale remains no greater than 1, and a positive record is formed that moves with the local displacement of the target entity.

[0166] The continuous length of the positive record is preferably 2 to 4 effective acquisition times, and more preferably 3 times; the 3 times form 2 continuous target relative background displacement processes, so that the source of the entity is based on a continuous co-displacement relationship, rather than spatial overlap at a single time.

[0167] Entity source synchronous verification propagation of continuous observations in the background outside the distant target.

[0168] Each candidate is verified sequentially according to the range of propagation location, the scale of propagation continuity, and the order of positions within the target area. If, within the same time period, independent propagation locations have formed identifiable separation, the target displacement relative to the background is established, and the time-varying visual components maintain continuous geometric correspondence, and the candidate observations on the far side of propagation cannot form a complete and continuous propagation relationship with the order of positions within the pre-propagation state and the target area, the pending visual evidence is registered as the source of the entity.

[0169] When multiple distant candidates for propagation still satisfy the requirement of a complete and continuous propagation relationship, pending visual evidence is used to continue to be added through subsequent observations.

[0170] The order of determining the source of an entity is fixed as follows: separation of independent propagation positions, displacement of the target relative to the background, continuous maintenance of the local geometric correspondence of the target, and verification of the connection of the propagation to the far side. The calculation of the next condition will only begin after the previous condition is met.

[0171] Table 4, Continuity parameters for source determination:

[0172]

[0173] Once the source is determined, the source determination module writes back the source status field in the same pending visual evidence based on the first entry event index.

[0174] When a source of light propagation is identified, continuous observations from the moment of first entry to the moment of leaving the target area are read, and the source status fields of these observations are uniformly updated to the source of light propagation. The corresponding collection time, spatial migration and propagation continuity records are retained.

[0175] When the entity source is determined, read the continuous observations written from the first entry time to the time the entity source is determined, and update the corresponding source status field to entity source. The time the entity source is determined is the last confirmation time when the independent propagation position forms a identifiable separation, the target displacement relative to the background is established, the local geometric correspondence of the target is maintained and the propagation far-side connection verification is completed.

[0176] Once the source of an entity is determined, newly added observations of the same event will continue to be registered along the established source.

[0177] After the source determination is completed, the corresponding historical observations that were previously in a pending state are updated according to the first entry event index to ensure that the source conclusion is consistent with the observation range of this first entry event.

[0178] The identity management module establishes a set of visual evidence for matching target identities.

[0179] This set consists of visual information of the target entity that was confirmed before the first entry event, as well as visual evidence that the source status field indicates the entity's origin.

[0180] For continuous observations where the source status field is entity source, the target identity matching visual evidence prioritizes using the local features of the target entity and the features of neighboring targets that were confirmed before the first entry, and updates the spatial position and local description value of these existing structures in the current frame. New visual structures that appear for the first time after the first entry and cannot be correlated with the target structures confirmed before the first entry through the target local geometric mapping are only stored in the observation record of the first entry event and are not directly written into the target identity matching visual evidence set because the event is determined to be entity source.

[0181] Continuous observations that are still pending or have been updated to sources of light propagation are retained in the corresponding first-entry event record; the identity matching process reads visual information from the target identity matching visual evidence set for target identity preservation and target identity recovery.

[0182] When performing target identity preservation, the identity management module extracts the local features of the target entity corresponding to the visual evidence of the historical entity source from the current target area, and uses the same type of target local geometric mapping as the initial entry stage to calculate the current corresponding position.

[0183] When optimizing similarity transformation, continue to use 3 to 5 sets, and further optimize 4 sets of stable point pairs; after mapping solution, calculate the back substitution residual of each point, and compare it with the historical normal geometric range corresponding to the local features of each target entity.

[0184] Once the current location is geometrically verified, the spatial location and local description value are updated using the target structure that is not currently covered by new pending visual evidence. If the corresponding location is temporarily covered by new time-varying visual components, the geometric location is updated using the nearby entity source structure, and the description value is based on the most recent valid observation that was not covered.

[0185] When the source of pending visual evidence is not yet determined and the target is temporarily lost, target detection and motion prediction form target candidates in the current video.

[0186] The identity management module reads the entity structure that has been used for target identity matching before the first entry event, as well as the visual evidence that has been identified as the source of the entity, and establishes a target local geometric mapping for each target candidate.

[0187] First, the local features of the historical target entity are positively mapped to the candidate current local position. Then, the corresponding points of the candidate are used to establish a reverse mapping and back-substitute to the local coordinates of the historical target. The residual between the back-substitute position and the historical position is calculated point by point, and the original adjacency order is checked.

[0188] When selecting similarity transformation, at least 3 sets of valid point pairs are obtained, and 4 sets are further selected. Only when the valid point pairs of the same candidate can solve the mapping together, and the back substitution residuals are all within the corresponding historical normal geometric range and the original adjacency order is maintained, the candidate is registered to satisfy the target local geometric correspondence.

[0189] When only one candidate in the target candidate set satisfies the complete target local geometric correspondence, the original target identity is associated with that candidate and the target identity is maintained.

[0190] When multiple candidates satisfy only a partial correspondence, their forward mapping, reverse substitution, and residual records are continued until subsequent continuous observations make the complete correspondence unique.

[0191] The target identity recovery continues to use the local features of the target entity and the historical normal geometric range used in the source determination stage, so as to keep the entity reference before entry, the source determination result and the geometric verification in the recovery stage consistent.

[0192] When the same target is affected by multiple time-varying visual components within the same time period, each time-varying visual component saves its pre-entry propagation state, local features of the target entity, and pending visual evidence according to its first entry time.

[0193] Both the background viewpoint compensation mapping and the target local geometry mapping are obtained by utilizing stable structures outside the coverage of the current pending visual evidence, thus avoiding the mixing of different pending events through compensation references.

[0194] Each pending visual evidence undergoes dual-relation calculation within the target area, propagation of subsequent observations to the far side, and source determination; visual evidence identified as the source of an entity is then incorporated into the identity matching record of the same target.

[0195] In cases of sampling interruption and temporary unavailability of mapping, the original timestamp of the corresponding acquisition time is retained and marked as an invalid observation.

[0196] The propagation state and state transition rate are formed only by adjacent valid observations within the same continuous valid segment; observations on both sides of the gap do not directly constitute adjacent propagation states; after regaining valid background viewpoint compensation and target local geometric mapping, recording continues from the new continuous valid segment.

[0197] When the time-varying visual component reaches the first entry judgment condition, the system only backtracks and reads the video frames that have been stored and whose acquisition time is earlier than the first entry time. These frames are used to complete the calculation of the pre-entry propagation state, local features of the target entity, and normal geometric range. If there are fewer than 5 effective observations before the first entry, the net migration distance does not reach the minimum identifiable migration amount, the pre-entry propagation direction does not pass the stability verification, or if effective target local geometric mapping and normal geometric range cannot be formed from the features of neighboring targets before the first entry, the first entry event and existing observations are saved as an unclosed source event and are not included in the source determination and identity access calculation of this event. Observations after the first entry are not added to the pre-entry history of this event.

[0198] When new time-varying visual components enter the target area from the background outside the target, the pre-entry propagation state, local features of the target entity, and pending visual evidence are re-established according to the new first entry event.

[0199] When multiple persistent compatible candidates exist on the far side of the propagation, pending visual evidence that has been formed and meets the prior reference conditions continues to be supplemented with subsequent valid observations until the subsequent observations can uniquely determine the corresponding subsequent observations.

[0200] According to the above processing, when the same time-varying visual component first enters the target area, it simultaneously preserves the independent propagation history before entry and the local history of the target entity corresponding to the entry position, and writes the continuous observations after entry into the same pending visual evidence.

[0201] The motion analysis module simultaneously calculates the continuous state of the entity's local geometric correspondence and entry forward propagation relationship for the same observation.

[0202] The source determination module determines the source of light propagation based on the complete and continuous propagation relationship formed at the far side of the propagation, and determines the source of the entity based on the geometric relationship that the entity still moves locally together with the target entity after being separated from the independent propagation position.

[0203] The identity management module writes the visual information that maintains a stable geometric correspondence with the existing target entity structure from the entity source observation into the target identity matching visual evidence set, which is used for target identity preservation and target identity recovery.

[0204] Each of the above stages is indexed by the same first entry event, so that the pre-entry history, observations during target coverage, propagation observations after departure, and observations of the target entity's own motion are continuously linked in the same event record.

[0205] The empirical quantiles in Tables 1 and 2 were obtained by sorting the corresponding historical samples according to the nearest rank method determined in this paper, without relying on the normal distribution assumption.

[0206] The continuous lengths set according to the effective acquisition time in Tables 1, 3, and 4 are all used to confirm the continuous relationship; the propagation speed, state transition rate, and propagation position recursion still use the original acquisition timestamp.

[0207] The above embodiments can be implemented in whole or in part by software, hardware, firmware or other arbitrary combinations. When implemented by software, the above embodiments can be implemented in whole or in part as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0208] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0209] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An intelligent underwater robot target detection system, characterized in that, include: The module includes video analytics, motion analysis, source determination, and identity management. The video analysis module acquires the target area in the underwater robot video, tracks the time-varying visual components that migrate from the background outside the target to the target area, retains the local features of the target entity in the pre-entry propagation state and the entry position when it first enters the target area, and associates the observations after entry as pending visual evidence. The motion analysis module determines the geometrically corresponding state based on the local features of the target entity, and determines the propagation direction and the order in which different positions within the target area are passed based on the pre-propagation state. The source determination module acquires subsequent observations of the background outside the target after crossing the target area along the propagation direction. When the propagation state, the sequence of observations, and subsequent observations form a continuous propagation relationship and the time-varying visual component is separated from the corresponding position of the local feature of the target entity, the source of the light propagation is determined. When the propagation position range determined based on the entry into the propagation state is separated from the corresponding position of the local feature of the target entity, the target is displaced relative to the background, and the time-varying visual component still maintains a geometric correspondence, and the continuous propagation relationship is excluded based on the observation of the background outside the target, the entity source is determined, and pending visual evidence is marked according to the determined source. The identity management module calls upon visual evidence of the target entity that has been confirmed before the time-varying visual component first enters the target area, as well as visual evidence of the entity's source, to perform target identity preservation and target identity restoration.

2. The intelligent underwater robot target detection system according to claim 1, characterized in that, The video analysis module tracks the time-varying visual components migrating from the background outside the target to the target area, and retains the specific operations for entering the forward propagation state as follows: Perform cross-frame correlation on continuous video observations in the background outside the target to form time-varying visual components; The propagation direction is determined by the continuous migration positions of the time-varying visual components before they first enter the target area, and the pre-entry propagation state is formed based on the position changes of adjacent observations. After the time-varying visual component first enters the target area, subsequent processing continuously calls the forward propagation state.

3. The intelligent underwater robot target detection system according to claim 2, characterized in that, The specific operations of the video analysis module to determine the local features of the target entity and form pending visual evidence are as follows: When the time-varying visual component first enters the target region, the local features of the target entity are determined from the target features that correspond to the entry position and have maintained cross-frame geometric correspondence before entry. By establishing a correlation between the pre-propagation state, the local features of the target entity, and the continuous observations after entering the target area, based on the same time-varying visual component, pending visual evidence is formed.

4. The intelligent underwater robot target detection system according to claim 1, characterized in that... The specific operations for determining the geometrically corresponding state in the motion analysis module are as follows: Establish a local geometric mapping of the target based on the local features of the target entity and the neighboring target features that are present across frames before the first entry into the target region. The continuous positions of time-varying visual components and local features of the target entity after entering the target region are mapped to the local coordinates of the target at the time of first entry, and the normal geometric range is determined based on the historical position residuals of the local features of the target entity before the first entry into the target region. The geometric correspondence state is determined based on the change between the current local offset of the time-varying visual component relative to the local features of the target entity and the initial local offset when it first enters the target entity. If the offset change is within the normal geometric range, the geometric correspondence state is maintained. If it is outside the normal geometric range, the time-varying visual component is determined to be separated from the corresponding position of the local features of the target entity.

5. The intelligent underwater robot target detection system according to claim 1, characterized in that, The specific operation of the motion analysis module in recording the order in which time-varying visual components pass through different locations within the target area is as follows: The propagation direction is determined based on the pre-propagation state, and the arrangement relationship of different positions within the target area is determined along the propagation direction; The time-varying visual components are recorded according to the video capture time, and the order in which they pass through each position after entering the target area is formed. After the time-varying visual component first enters the target area, subsequent processing continues to use the propagation direction.

6. The intelligent underwater robot target detection system according to claim 1, characterized in that, The specific operation of the source determination module in obtaining subsequent observations of the background outside the target after crossing the target area along the propagation direction is as follows: The propagation location range under independent propagation conditions is determined based on the pre-propagation state and video acquisition time. Continuous observations are extracted from the background outside the target area after the propagation direction crosses the target area. Observations that match the range of the propagation location are selected based on temporal continuity and spatial migration continuity. Single observations that continuously satisfy the matching relationship are identified as subsequent observations.

7. The intelligent underwater robot target detection system according to claim 6, characterized in that, The specific operation of the source determination module to determine the source of light propagation is as follows: The video acquisition time is used to determine the pre-propagation state, sequence, and subsequent observations. The resulting continuous propagation relationship passes sequentially through the background outside the target, the target area, and the background outside the target after crossing the target area along the propagation direction. When the time-varying visual component is separated from the corresponding position of the local feature of the target entity and the local feature of the target entity maintains cross-frame geometric correspondence within the target area, the pending visual evidence is determined as the source of light propagation, and the light propagation source marker is associated with the corresponding observation formed after the time-varying visual component first enters the target area.

8. The intelligent underwater robot target detection system according to claim 1, characterized in that, The specific operations for determining the source of an entity by the source determination module are as follows: The propagation location range under independent propagation conditions is determined based on the pre-propagation state and video acquisition time. After the propagation location range is separated from the corresponding location of the local features of the target entity, the displacement of the target relative to the background is detected; During the target's displacement relative to the background, the time-varying visual components still maintain a geometric correspondence. Propagation continuity verification is performed on continuous observations in the background outside the target after crossing the target area along the propagation direction. When the verification result excludes the continuous propagation relationship between continuous observations and the propagation state and sequence before entry, the pending visual evidence is identified as the entity source, and the entity source marker is associated with the corresponding observation.

9. The intelligent underwater robot target detection system according to claim 1, characterized in that, The identity management module invokes visual evidence from the entity source to perform target identity preservation and target identity restoration operations as follows: Once the pending visual evidence is identified as the source of the entity, visual information that maintains cross-frame geometric correspondence with the local features of the target entity is determined from the corresponding observations, and this visual information is marked as visual evidence of the entity source for target identity matching.

10. The intelligent underwater robot target detection system according to claim 9, characterized in that, The specific operations performed by the identity management module to restore the target identity during the period when the source of pending visual evidence is undetermined are as follows: Acquire target candidates detected after the target is temporarily lost; Only visual evidence used for target identity matching and visual evidence already identified as the source of the entity before the time-varying visual component first enters the target region are invoked to establish a local geometric mapping of the target candidate; Identify target candidates that form a unique geometric correspondence with the local features of the target entity, and restore the target identity of the target candidates.

Images