River channel pedestrian behavior identification method and device, equipment and medium

Abstract

The invention provides a river channel pedestrian behavior identification method and device, equipment and a medium, and relates to the technical field of data processing. The river channel pedestrian behavior identification method comprises the following steps: acquiring video stream data in a target river channel scene; performing pedestrian detection and pedestrian tracking on the video stream data, and determining a pedestrian trajectory image sequence corresponding to a target pedestrian; determining weight scores corresponding to different images in the pedestrian track image sequence, and determining a multi-channel image sequence corresponding to a target pedestrian according to the weight scores; converting the multi-channel image sequence into a grayscale image sequence, and performing feature extraction and feature fusion on the multi-channel image sequence and the grayscale image sequence to obtain fusion features; and inputting the fusion feature into a pre-trained visual language model to obtain a behavior classification corresponding to the target pedestrian. According to the technical scheme, the efficiency and the accuracy of river channel pedestrian behavior recognition can be improved.

Description

Background Art

[0002] River safety management is crucial to preventing drowning accidents and maintaining social security. Traditional identification of pedestrian behavior in rivers mainly relies on manual inspections or video surveillance. However, manual inspections are limited by the number and energy of inspectors, making it difficult to monitor the river at all times and in all areas. Traditional video surveillance technology is difficult to intelligently identify and classify pedestrian behavior and requires manual judgment, which is inefficient and prone to missed judgments. Therefore, there is still room for improvement in the efficiency and accuracy of the identification method of pedestrian behavior in rivers.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute relevant technology known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the embodiments of the present disclosure is to provide a method for identifying pedestrian behavior in a river, an apparatus for identifying pedestrian behavior in a river, an electronic device and a computer-readable storage medium, thereby improving the efficiency and accuracy of identifying pedestrian behavior in a river, at least to a certain extent.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for identifying pedestrian behavior in a river is provided, the method comprising: acquiring video stream data in a target river scene; performing pedestrian detection and pedestrian tracking on the video stream data to determine a pedestrian trajectory image sequence corresponding to the target pedestrian; determining weight scores corresponding to different images in the pedestrian trajectory image sequence, and determining a multi-channel image sequence corresponding to the target pedestrian based on the weight scores; converting the multi-channel image sequence into a grayscale image sequence, and performing feature extraction and feature fusion on the multi-channel image sequence and the grayscale image sequence to obtain fused features; inputting the fused features into a pre-trained visual language model to obtain a behavior classification corresponding to the target pedestrian.

[0007] In some example embodiments of the present disclosure, based on the aforementioned scheme, the video stream data is subjected to pedestrian detection and pedestrian tracking to determine a pedestrian trajectory image sequence corresponding to a target pedestrian, including: preprocessing images in the video stream data to obtain preprocessed video stream data; performing pedestrian detection on the preprocessed video stream data to obtain a first target detection frame of the target pedestrian in the current image; predicting the position of the target pedestrian in the next frame of the image using a Kalman filter to obtain a predicted detection frame; matching a second target detection frame of the target pedestrian in the next frame of the image with the predicted detection frame, and determining a corresponding relationship between the second target detection frame and the target pedestrian based on the matching result; repeating the pedestrian detection and matching process to obtain a pedestrian trajectory image sequence corresponding to the target pedestrian.

[0008] In some example embodiments of the present disclosure, based on the aforementioned scheme, the pedestrian detection and matching process are repeated to obtain a pedestrian trajectory image sequence corresponding to the target pedestrian, including: repeating the pedestrian detection and matching process to determine a detection frame set corresponding to the target pedestrian; performing non-maximum suppression processing on the detection frame set to obtain a target detection frame group; and determining a pedestrian trajectory image sequence corresponding to the target pedestrian based on the target detection frame group.

[0009] In some example embodiments of the present disclosure, based on the aforementioned scheme, determining the weight scores corresponding to different images in the pedestrian trajectory image sequence, and determining the multi-channel image sequence corresponding to the target pedestrian according to the weight scores, includes: performing pixel difference calculations on the current image and adjacent frame images in the pedestrian trajectory image sequence to obtain pixel difference values; determining the weight scores according to the pixel difference values, and determining the multi-channel image sequence according to images whose weight scores are not less than a weight threshold.

[0010] In some example embodiments of the present disclosure, based on the aforementioned scheme, the feature fusion of the multi-channel image sequence and the grayscale image sequence to obtain a fused feature includes: using a convolutional neural network to extract features of the multi-channel image sequence to obtain a first image feature, the first image feature includes one or more of a color feature, a texture feature, and a light intensity feature; based on a local feature description operator, feature extraction of the grayscale image sequence to obtain a second image feature, the second image feature includes one or more of a structural feature, an edge feature, and a corner feature; splicing the first image feature and the second image feature to obtain a spliced ​​feature; determining the feature weights corresponding to different channels in the spliced ​​feature through an attention module, and weighted combining the features corresponding to different channels based on the feature weights to obtain the fused feature.

[0011] In some example embodiments of the present disclosure, based on the aforementioned scheme, the method for identifying pedestrian behavior in a river further includes: in response to classification of the behavior corresponding to the target pedestrian as falling into water, acquiring an environmental image of the current location of the target pedestrian; and determining a target rescue path based on road information in the environmental image.

[0012] In some example embodiments of the present disclosure, based on the aforementioned scheme, the method for identifying pedestrian behavior in a river further includes: in response to a result of the pedestrian detection being that the target pedestrian is located in the river, acquiring a river image of the location of the target pedestrian; and determining a behavior classification corresponding to the target pedestrian based on a water ripple pattern in the river image.

[0013] According to a second aspect of an embodiment of the present disclosure, a device for identifying pedestrian behavior in a river is provided, and the device includes: a video data acquisition module, used to acquire video stream data in a target river scene; a pedestrian trajectory determination module, used to perform pedestrian detection and pedestrian tracking on the video stream data, and determine a pedestrian trajectory image sequence corresponding to the target pedestrian; a weight score determination module, used to determine the weight scores corresponding to different images in the pedestrian trajectory image sequence, and determine a multi-channel image sequence corresponding to the target pedestrian based on the weight scores; a feature fusion module, used to convert the multi-channel image sequence into a grayscale image sequence, and perform feature fusion on the multi-channel image sequence and the grayscale image sequence to obtain a fused feature; and a behavior classification module, used to input the fused feature into a pre-trained visual language model to obtain a behavior classification corresponding to the target pedestrian.

[0014] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and the computer-readable instructions, when executed by the processor, implement the above-mentioned method for identifying pedestrian behavior in a river.

[0015] According to a fourth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for identifying pedestrian behavior in a river channel described above is implemented.

[0016] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0017] The method for identifying pedestrian behavior in a river channel in the example embodiment of the present disclosure, on the one hand, performs pedestrian detection and tracking on video stream data, determines the pedestrian trajectory image sequence corresponding to the target pedestrian, can accurately identify pedestrians in complex backgrounds, and can monitor the individual's motion path, and can maintain continuity even when the number of pedestrians changes or there is occlusion; on the other hand, by evaluating the quality of each frame in the pedestrian trajectory image sequence, the clearest and most recognizable multi-channel image sequence can be selected for subsequent analysis to improve recognition accuracy; in addition, the feature fusion of multi-channel images and grayscale images allows the model to capture information about pedestrian behavior from different angles, enhancing the robustness and recognition ability of the model; on the other hand, using the visual language model, it is possible to perform efficient behavior classification based on the fused features to identify whether pedestrians are performing dangerous activities in the river channel area. Thereby, the efficiency and accuracy of identifying pedestrian behavior in the river channel are improved, which is conducive to rapid response to emergencies.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0020] Figure 1 A flowchart of a method for identifying pedestrian behavior in a river according to some embodiments of the present disclosure is schematically shown.

[0021] Figure 2 The following is a schematic diagram showing the recognition process of a recognition model for pedestrian behavior in a river according to some embodiments of the present disclosure.

[0022] Figure 3 A block diagram of a device for identifying pedestrian behavior in a river according to some embodiments of the present disclosure is schematically shown.

[0023] Figure 4 A schematic diagram of the structure of a computer system of an electronic device according to some embodiments of the present disclosure is schematically shown.

[0024] Figure 5 A schematic diagram of a computer-readable storage medium according to some embodiments of the present disclosure is schematically shown.

[0025] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0027] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0028] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concept of the example embodiments to those skilled in the art.

[0030] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0031] Furthermore, the drawings are only schematic illustrations and are not necessarily drawn to scale. The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0032] River safety management is a key link in ensuring public safety and maintaining social stability, especially in preventing drowning accidents. Traditional methods rely heavily on human inspections, which are limited by the number and physical strength of inspectors and cannot guarantee all-weather and all-round coverage of rivers. They are inefficient and leave safety hazards. At the same time, although conventional video surveillance methods can record on-site images, they lack intelligent analysis capabilities. The work of identifying and classifying pedestrian behavior still requires manual intervention, which is not only inefficient, but also prone to omissions of key events due to human negligence.

[0033] In order to solve all or part of the technical problems in the above-mentioned related technologies, in an exemplary embodiment of the present disclosure, a method for identifying pedestrian behavior in a river is firstly proposed. Figure 1 The following is a schematic diagram of a flow chart of a method for identifying pedestrian behavior in a river according to some embodiments of the present disclosure. Figure 1 As shown, the method for identifying pedestrian behavior in a river may include the following steps:

[0034] Step S110, obtaining video stream data of a target river scene;

[0035] Step S120, performing pedestrian detection and pedestrian tracking on the video stream data to determine a pedestrian trajectory image sequence corresponding to the target pedestrian;

[0036] Step S130, determining weight scores corresponding to different images in the pedestrian trajectory image sequence, and determining a multi-channel image sequence corresponding to the target pedestrian according to the weight scores;

[0037] Step S140, converting the multi-channel image sequence into a grayscale image sequence, and performing feature extraction and feature fusion on the multi-channel image sequence and the grayscale image sequence to obtain fusion features;

[0038] Step S150: input the fused features into a pre-trained visual language model to obtain a behavior classification corresponding to the target pedestrian.

[0039] The above-mentioned method for identifying pedestrian behavior in a river first captures the real-time video stream of the target river area, which constitutes the basic data source of the entire pedestrian behavior recognition process. Then, the video stream is analyzed, and all pedestrians are identified using the pedestrian detection algorithm. The continuous trajectory of each pedestrian is determined by the target tracking technology to form a pedestrian trajectory image sequence. Secondly, in the pedestrian trajectory image sequence, a weight score is assigned to each frame of the image, and the high-quality image that best represents the target pedestrian is selected based on the score to form a multi-channel image sequence. Then, the multi-channel image sequence is converted into a grayscale image sequence, and feature extraction is performed on the two groups of image sequences at the same time, and then the extracted features are fused to generate a more comprehensive fusion feature. Finally, the fusion feature is input into a pre-trained visual language model, and the model will analyze the input fusion feature and output the behavior category of the target pedestrian, such as "staying", "crossing the boundary" or "falling into the river", etc., thereby providing technical support for rapid response to emergencies.

[0040] Next, the method for identifying the behavior of pedestrians in a river in the above exemplary embodiment will be further described.

[0041] In step S110, video stream data of a target river scene is obtained.

[0042] The target river scene may represent a specific river area that needs to be monitored by video, such as the river around a hydropower station, the river bank, or the geographical area near a cross-river bridge, and other river areas that need to pay attention to pedestrian activities. Video stream data may represent real-time or pre-recorded video signals from a camera or other video acquisition device, which are presented in the form of a series of continuous image frames, each of which represents the state of the target river scene at a certain moment. By acquiring video stream data under the target river scene, real-time monitoring of the river area can be ensured, providing the necessary raw data for subsequent pedestrian detection, pedestrian tracking, and behavior recognition.

[0043] In step S120, pedestrian detection and pedestrian tracking are performed on the video stream data to determine a pedestrian trajectory image sequence corresponding to the target pedestrian.

[0044] Among them, pedestrian detection can refer to the process of locating and identifying pedestrians in video stream data. Pedestrian tracking can refer to the process of continuously identifying and tracking the same pedestrian in consecutive video frames. Pedestrian trajectory image sequence can refer to a series of consecutive image frames that contain the movement path of a specific pedestrian in a video. Pedestrian trajectory image sequence provides a continuous view of the target pedestrian's movements and position changes.

[0045] In some embodiments, pedestrian detection and pedestrian tracking are performed on video stream data to determine a pedestrian trajectory image sequence corresponding to a target pedestrian, specifically including the following steps: preprocessing images in the video stream data to obtain preprocessed video stream data; performing pedestrian detection on the preprocessed video stream data to obtain a first target detection frame of the target pedestrian in the current image; using a Kalman filter to predict the position of the target pedestrian in the next frame of the image to obtain a predicted detection frame; matching the second target detection frame of the target pedestrian in the next frame of the image with the predicted detection frame, and determining the correspondence between the second target detection frame and the target pedestrian based on the matching result; repeating the pedestrian detection and matching process to obtain a pedestrian trajectory image sequence corresponding to the target pedestrian.

[0046] Among them, the Kalman filter is a recursive linear least squares estimator that can be used for state estimation of dynamic systems (such as systems that change over time). In the presence of measurement noise and process noise, the Kalman filter can effectively predict and update the optimal estimate of the system state, thereby predicting the possible position of the pedestrian in the next frame of the image.

[0047] Specifically, the YOLOv8 (You Only Look Once version 8) detector can be used to detect pedestrians in the preprocessed image. YOLOv8 is a real-time target detection framework that can detect the position and category of all pedestrian targets by performing a single inference on a single image. Through YOLOv8, the first target detection frame of the target pedestrian in the current image can be obtained. Each frame represents a potential pedestrian target, and each detection frame has its corresponding confidence score. Then, based on the position and speed of the pedestrian detection frame in the previous frame, the Kalman filter predicts the possible position of the pedestrian in the next frame to obtain the predicted detection frame. Next, the second target detection frame of the target pedestrian in the next frame image is matched with the predicted detection frame. Specifically, cascade matching is performed first, that is, the second target detection frame with the highest confidence score is cascade matched with the predicted detection frame. If the match is successful, the subsequent steps are performed. If the match fails, the intersection over union (IOU) matching is performed to calculate the intersection over union ratio of the two frames. If the IOU is within the preset threshold, the two frames are considered to be matched successfully, otherwise they are considered to be different targets. As shown in the following formula (1), A represents the predicted detection frame, B represents the second target detection frame, the intersection between the two is used as the numerator, and the union is used as the denominator. The IoU is between [0,1], and 1 indicates that the predicted frame and the detection frame overlap, and 0 indicates that there is no relationship between the two.

[0048]

[0049] Finally, the pedestrian trajectory is determined, and the above pedestrian detection and matching process is repeated. For each video stream data, pedestrian detection, prediction and matching are continuously performed until the end of the video stream. For each target pedestrian, the corresponding pedestrian trajectory image sequence can be obtained.

[0050] In some embodiments, the images in the video stream data are preprocessed. Specifically, an image denoising algorithm may be used to remove noise in the video image to improve image quality, thereby improving the accuracy of pedestrian detection and tracking. An image illumination correction algorithm may also be used to adjust image brightness to reduce the impact of illumination changes on pedestrian detection and tracking. Image enhancement algorithms, such as sharpening and contrast enhancement, are used to enhance image features and improve the robustness of pedestrian detection and tracking.

[0051] Exemplarily, continuous video stream data can be converted into discrete image data for frame-by-frame analysis. Then, the image size is unified to eliminate the impact of resolution differences and facilitate the processing of subsequent models. For example, the size of each frame of the image can be scaled to 640×640. If the short side of the image is not enough to scale to 640 pixels, 0 pixels are used for padding to keep its aspect ratio unchanged and reach a size of 640×640. Next, the pixel values ​​of the image are normalized. Specifically, the mean and variance of all pixel values ​​of each frame of the image are calculated, the mean is subtracted from each pixel value, and then divided by the variance to obtain the normalized pixel value.

[0052] In some embodiments, the pedestrian detection and matching process is repeated to obtain a pedestrian trajectory image sequence corresponding to the target pedestrian, specifically including the following steps: repeating the pedestrian detection and matching process to determine a detection frame set corresponding to the target pedestrian; performing non-maximum suppression processing on the detection frame set to obtain a target detection frame group; and determining a pedestrian trajectory image sequence corresponding to the target pedestrian based on the target detection frame group.

[0053] Among them, non-maximum suppression (NMS) can be used to compare the confidence scores of detection boxes and eliminate highly overlapping detection boxes, so as to select the detection box that is most likely to represent the real target.

[0054] Specifically, through pedestrian detection and Kalman filter prediction, a detection frame set for each pedestrian can be established according to the pedestrian detection results in each frame, and each detection frame contains the location information and pedestrian ID of the pedestrian. Then, non-maximum suppression is performed on each detection frame in the detection frame set, and the detection frame with the highest confidence score is selected as the current best detection frame. The IoU between this detection frame and all other detection frames in the set is calculated. If the IoU between other detection frames and the current best detection frame exceeds a preset threshold, these detection frames are considered to overlap. For example, the preset threshold can be 0.5, 0.6 or other suitable values, and it is removed from the detection frame set to avoid repeated detection of the same pedestrian. After removing the detection frame that overlaps with the current best detection frame, the detection frame with the highest confidence score is selected again from the remaining detection frames, and the non-maximum suppression process is repeated until all detection frames are subjected to non-maximum suppression processing to obtain a target detection frame group. Finally, based on the target detection frame group, the continuous position information of each pedestrian in the video stream can be determined. For each pedestrian ID, all detection frames related to the pedestrian in the target detection frame group are collected, and the image frames corresponding to these detection frames constitute the pedestrian trajectory image sequence of the pedestrian.

[0055] Furthermore, the YOLOv8 detector and Kalman filter prediction are used in the above pedestrian detection and matching process to ensure accurate identification of pedestrians even in complex backgrounds or occlusions. And through cascade matching and IoU matching, different individuals can be effectively distinguished and tracked, reducing false detections and missed detections. In addition, non-maximum suppression processing can eliminate the redundancy caused by multiple detection frames covering the same pedestrian, improve the quality of the detection frame set, and reduce the amount of calculation for subsequent processing.

[0056] In step S130, weight scores corresponding to different images in the pedestrian trajectory image sequence are determined, and a multi-channel image sequence corresponding to the target pedestrian is determined according to the weight scores.

[0057] Among them, the weight score can be used to evaluate the importance of different images in the pedestrian trajectory image sequence, and finally determine the multi-channel image sequence corresponding to the target pedestrian. Exemplarily, the confidence score of the detection box corresponding to the target pedestrian in the image can be used as the weight score, the difference between the image and its adjacent image frames can be used as the weight score, and the feature significance of the image, such as color, texture, posture features, etc., can also be used as the weight score. Of course, in other embodiments of the present disclosure, the weight score corresponding to the image can also be determined by other suitable methods.

[0058] A multi-channel sequence image can represent an image sequence consisting of images containing multiple color channels, that is, multi-channel images. The multi-channel images can be RGB images, HSV images, YUV images, Lab images, and other images with multiple color channels.

[0059] In some embodiments, weight scores corresponding to different images in a pedestrian trajectory image sequence are determined, and a multi-channel image sequence corresponding to a target pedestrian is determined based on the weight scores, specifically including the following steps: performing pixel difference calculations on a current image and an adjacent frame image in the pedestrian trajectory image sequence to obtain a pixel difference value; determining a weight score based on the pixel difference value, and determining a multi-channel image sequence based on images whose weight scores are not less than a weight threshold.

[0060] Specifically, for each frame in the pedestrian trajectory image sequence, it is compared pixel by pixel with the adjacent frame image, the difference value of each pixel is calculated, and the difference values ​​of all pixels are summed to obtain the total pixel difference value between the current image and the adjacent frame image. Then, the total pixel difference value is used as the weight score corresponding to the current image, and the images with weight scores less than the weight threshold are eliminated, and the images with weight scores not less than the weight threshold are selected as the components of the multi-channel image sequence.

[0061] In step S140, the multi-channel image sequence is converted into a grayscale image sequence, and feature extraction and feature fusion are performed on the multi-channel image sequence and the grayscale image sequence to obtain fusion features.

[0062] Among them, the grayscale image sequence can represent a sequence composed of a series of grayscale images, in which each frame of the image contains only one color channel. The fusion feature is the feature obtained by fusing the features extracted from the multi-channel image sequence and the grayscale image sequence. The fusion feature contains the information of these two image types and can more comprehensively describe the image content, thereby improving the recognition effect of the model.

[0063] For example, when the images in the multi-channel image sequence are RGB images, the multi-channel image sequence can be converted into a grayscale image sequence by the following method (2):

[0064] gray = 0.299 × R + 0.587 × G + 0.114 × B (2)

[0065] Among them, gray represents the converted grayscale image, and R, G, and B represent the red, green, and blue channels in the RGB image respectively.

[0066] In some embodiments, features corresponding to multi-channel images and grayscale images can be extracted by edge detection, corner detection, color histogram and other methods, and then feature fusion of the multi-channel image sequence and the grayscale image sequence is performed by ω=λω1+(1-λ)ω2, where ω represents fusion features, ω1 represents multi-channel image features, ω2 represents grayscale image features, and λ is an adjustment parameter whose value range is [0,1].

[0067] In some embodiments, feature fusion is performed on a multi-channel image sequence and a grayscale image sequence to obtain a fused feature, which specifically includes the following steps: using a convolutional neural network to extract features from the multi-channel image sequence to obtain a first image feature, the first image feature includes one or more of a color feature, a texture feature, and a light intensity feature; based on a local feature description operator, feature extraction is performed on the grayscale image sequence to obtain a second image feature, the second image feature includes one or more of a structural feature, an edge feature, and a corner feature; the first image feature and the second image feature are spliced ​​to obtain a spliced ​​feature; the feature weights corresponding to different channels in the spliced ​​feature are determined through an attention module, and the features corresponding to different channels are weighted combined based on the feature weights to obtain a fused feature.

[0068] Among them, the first image feature can represent the feature set extracted after processing the multi-channel image sequence through a convolutional neural network. Color features can be used to describe the distribution, combination and change of colors in an image. Texture features can be used to describe the visual performance and spatial arrangement of textures in an image. Light intensity features can be used to reflect the light intensity information of pixels in an image, including brightness and contrast. The second image feature can represent the feature set extracted after processing the grayscale image sequence through a local feature description operator. Structural features can be used to describe the geometric structure and shape information of the target object in the image. Edge features can be used to describe the boundary information of objects in an image or the information of the edge of a region. Corner features can be used to describe the position of the corners or significant corners of the target object in the image.

[0069] Of course, in other embodiments of the present disclosure, the first image features may also include spatial relationship features, scale features, local area features, and other categories of features that can describe other features of multi-channel images, and the second image features may also include spot features, gradient features, histogram features, and other categories of features that can describe other features of grayscale images.

[0070] Specifically, first, a convolutional neural network is used to extract features from a multi-channel image sequence, and feature information corresponding to the image, such as color distribution, color texture, illumination information, color contrast, and color space, is obtained, and used as the first image feature. Then, based on local feature description operators such as SIFT, SURF, or ORB, feature extraction is performed on the grayscale image sequence to obtain various features such as structural shape, identify key points and edges in the image, capture the shape and contour information of the target pedestrian, and generate the second image feature. Secondly, the first and second features are stacked vertically according to the channel dimension. Assuming that the first feature is C and the second feature is S, the spliced ​​feature is F = [C; S]. Thirdly, the spliced ​​feature is input into the attention module composed of a multi-layer perceptron and a self-attention mechanism, and a weight value is generated for each channel in the spliced ​​feature. The weight value reflects the contribution of each channel to the final feature representation, and the weight value is used as the feature weight corresponding to different channels. Finally, based on the feature weight, the different channels of the spliced ​​feature are weighted, and the weighted feature is the fusion feature. Through the above process, the fused features not only contain the color, structure, texture, edge, light intensity, corner point and other information of the original image, but also emphasize the most relevant and discriminative features through the attention mechanism, so that the model can adaptively adjust the importance of different features, thereby improving the recognition accuracy and robustness.

[0071] In step S150, the fused features are input into a pre-trained visual language model to obtain a behavior classification corresponding to the target pedestrian.

[0072] Among them, the visual language model can be built based on the Llama3-8B model. The Llama3-8B model is a pre-trained language model based on the Transformer architecture, which can be used for image data processing and has a good feature processing effect. The behavior classification can include pedestrian behaviors that may occur in other river areas, such as falling into the water, staying, crossing the boundary, going into the river, running, fishing, taking pictures, picnicking, swimming, walking the dog, etc. In other embodiments of the present disclosure, the visual language model can also be constructed based on other suitable network models such as LeNet network, VGG network and residual (ResNet) network.

[0073] In some embodiments, the method for identifying pedestrian behavior in a river further includes: in response to the behavior corresponding to the target pedestrian being classified as falling into water, obtaining an environmental image of the target pedestrian's current location; and determining a target rescue path based on road information in the environmental image.

[0074] The environmental image may refer to a visual image of the surrounding environment of the river area where the target pedestrian is located, obtained by a camera, drone or other device. The road information may refer to the road-related data that can be identified and analyzed in the environmental image, which can be used to describe the network structure, attributes and status of the road. The target rescue path may refer to the optimal or appropriate rescue route planned from the rescue starting point to the location of the target pedestrian based on the road information in the environmental image.

[0075] Specifically, according to the real-time acquired video stream data, when the behavior of the target pedestrian is classified as falling into the water, the rescue process is triggered. Then, the target shooting device that collects the video stream data is determined, and according to the location information of the target shooting device in the river area, all the shooting devices in its neighboring area are determined, wherein the neighboring area can represent a circular area with a radius of 200 meters, 400 meters or 800 meters or other appropriate radius centered on the target shooting device. The environmental images taken by all the shooting devices in the neighboring area are retrieved, and the road information in the environmental images, such as road boundaries, obstacles, traffic signs, etc., are extracted using convolutional neural networks, support vector machines, target detection and other suitable image recognition methods. Then, according to the extracted road information, a road topology map is constructed, and the path planning algorithm is used to calculate the optimal rescue path from the rescue starting point to the location of the target pedestrian, ensuring that the rescue path avoids obstacles and meets the rescue speed and safety requirements. Finally, the planned rescue path is displayed to the rescue personnel in a visual form to achieve rapid and accurate rescue of the target pedestrian who fell into the water and improve the rescue success rate.

[0076] In some embodiments, the above-mentioned method for classifying pedestrian behavior in a river also includes: in response to the result of pedestrian detection that the target pedestrian is located in the river, obtaining a river image of the target pedestrian's location; and determining a behavior classification corresponding to the target pedestrian based on a water ripple pattern in the river image.

[0077] Among them, the water ripple pattern can represent the specific form and dynamic characteristics of the ripples on the water surface. For example, the water ripple pattern can include other forms of ripples such as circular ripples, linear ripples, V-shaped ripples, shock ripples, vortices, etc. Among them, circular ripples can represent circular ripples that spread outward from a center point. Linear ripples can represent straight or curved ripples, which have a clear linear direction, consistent with the direction of human body movement. V-shaped ripples can represent V-shaped ripples that spread from a specific position to both sides. Shock ripples can represent strong shock ripples caused by a person falling into the water. Vortex can represent rotating ripples caused by eddies in the water flow.

[0078] Different water ripple patterns may correspond to different behavior classifications. For example, the behavior classification corresponding to circular ripples and vortices may be falling into water, the behavior classification corresponding to linear ripples and V-shaped ripples may be swimming, and the behavior classification corresponding to impact ripples may be diving.

[0079] Furthermore, a plurality of river images containing pedestrians may be acquired in advance, and then these river images may be annotated according to their corresponding water ripple patterns to obtain sample data consisting of river images and water ripple pattern labels. Then, the initial water ripple pattern classification model may be trained according to the sample data to obtain a trained water ripple pattern classification model. The water ripple pattern classification model may be constructed based on other suitable network models such as convolutional neural networks, recurrent neural networks, and support vector machines. After obtaining the river image at the location of the target pedestrian, the river image may be input into the water ripple pattern classification model to obtain the water ripple pattern corresponding to the river image, and then the behavior classification corresponding to the target pedestrian may be determined according to the water ripple pattern.

[0080] The disclosed embodiment also provides a classification model for the behavior of pedestrians in a river. Figure 2 The following is a schematic diagram showing the recognition process of the river pedestrian behavior recognition model. Figure 2 As shown, the recognition model of pedestrian behavior in a river includes a pedestrian detection network, a pedestrian tracking network, a feature fusion network, and a visual language model. Specifically, the video stream data in the river scene is input into the recognition model of pedestrian behavior in the river, and then the first video frame, the second video frame, the nth video frame, and other video frames in the video stream data are preprocessed by filling processing, noise removal, and other preprocessing to obtain preprocessed video frames. Then, the pedestrian detection network and the pedestrian tracking network are used to perform pedestrian detection on the preprocessed video stream data, and the first target detection frame of the target pedestrian in the current image is obtained. The position of the target pedestrian in the next frame image is predicted by the Kalman filter to obtain the predicted detection frame, and the second target detection frame of the target pedestrian in the next frame image is matched with the predicted detection frame, and the corresponding relationship between the second target detection frame and the target pedestrian is determined according to the matching result, and the pedestrian detection and matching process is repeated to determine the detection frame set corresponding to the target pedestrian, and the detection frame set is subjected to non-maximum suppression processing to obtain the target detection frame group, and the pedestrian trajectory image sequence corresponding to the target pedestrian is determined based on the target detection frame group. Next, the weight scores corresponding to different images in the pedestrian trajectory image sequence are determined, and the multi-channel image sequence corresponding to the target pedestrian is determined based on the weight scores. Secondly, the multi-channel image sequence is converted into a grayscale image sequence using a feature fusion network, and the multi-channel image sequence and the grayscale image sequence are fused to obtain fused features. Finally, the fused features are input into the pre-trained visual language model to obtain the behavior classification corresponding to the target pedestrian.

[0081] The following is a detailed explanation of the training process of the above-mentioned river pedestrian behavior classification model.

[0082] Different video stream data are collected in a specific river area. The video stream data includes multiple pedestrian behaviors. The classification labels corresponding to different video stream data are determined according to the pedestrian behaviors. The classification labels may include falling into the water, staying, crossing the boundary, going into the river, running, fishing, taking pictures, picnicking, swimming, walking the dog, etc. The video stream data and classification labels are used as the training data set of the model. Then, the video stream data is preprocessed, and the preprocessed video stream data is used to detect pedestrians using the YOLOv8 detector to identify pedestrians in the image and obtain their target detection frames. The pedestrians in the image are found and a unique ID number is assigned to each pedestrian for subsequent tracking and classification. The detected pedestrian target frame is predicted by Kalman filtering to predict the area where the pedestrian may appear in the next frame. The Kalman prediction frame is matched with the detection frame of the current frame, and it is determined whether it is the same target based on the matching result. If the match is successful, the target is continued to be tracked; if the match fails, the ID number is reassigned and regarded as a new target. The motion trajectory of the pedestrian is tracked, and the same pedestrian is identified in consecutive frames to obtain a sequence image of the pedestrian trajectory.

[0083] In addition, the pedestrian position predicted by the model is compared with the pre-labeled sample position, and the loss function Focal Loss is calculated. The learning rate is adjusted according to the loss value, and iterative training is performed to adjust the accuracy of the pedestrian detection network and the pedestrian tracking network. The specific formula of Focal Loss is shown in the following formula (3):

[0084]

[0085] Among them, p t represents the predicted probability of the target pedestrian position, r represents the adjustment parameter, logp t represents the cross entropy loss function, which is used to measure the difference between the predicted position and the sample position.

[0086] Next, the weight scores corresponding to different images in the pedestrian trajectory image sequence are determined, and the multi-channel image sequence corresponding to the target pedestrian is determined according to the weight scores. The multi-channel image sequence is converted into a grayscale image sequence, and the multi-channel image sequence and the grayscale image sequence are feature fused to obtain the fused feature.

[0087] Finally, the fused features are input into the initial visual language model, and the classification results of the model are used as training labels until the number of iterations of the initial visual language model reaches the maximum number or the loss function converges, thereby realizing the model training of the initial visual language model and obtaining the target visual language model.

[0088] The method for identifying pedestrian behavior in a river channel in the example embodiment of the present disclosure, on the one hand, performs pedestrian detection and tracking on video stream data, determines the pedestrian trajectory image sequence corresponding to the target pedestrian, can accurately identify pedestrians in complex backgrounds, and can monitor the individual's motion path, and can maintain continuity even when the number of pedestrians changes or there is occlusion; on the other hand, by evaluating the quality of each frame in the pedestrian trajectory image sequence, the clearest and most recognizable multi-channel image sequence can be selected for subsequent analysis to improve recognition accuracy; in addition, the feature fusion of multi-channel images and grayscale images allows the model to capture information about pedestrian behavior from different angles, enhancing the robustness and recognition ability of the model; on the other hand, using the visual language model, it is possible to perform efficient behavior classification based on the fused features to identify whether pedestrians are performing dangerous activities in the river channel area. Thereby, the efficiency and accuracy of identifying pedestrian behavior in the river channel are improved, which is conducive to rapid response to emergencies.

[0089] Furthermore, by preprocessing the video stream data and using the Kalman filter to predict the position of pedestrians, the accuracy of pedestrian detection can be improved, and stable tracking can be maintained even in complex backgrounds. The application of the non-maximum suppression method reduces the redundancy of the detection frame and avoids the problem of the same pedestrian being marked multiple times, thereby improving the detection efficiency. By calculating the pixel difference value and assigning a weight score, image frames that are more representative of the behavior can be screened out to form a multi-channel image sequence, which helps to highlight key features. The first image feature is combined with the second image feature, and the attention mechanism is used to weight different features, which enhances the robustness and accuracy of the model for behavior recognition. In addition, after the target pedestrian is identified as falling into the water, the target rescue path is quickly determined based on the environmental image of the pedestrian's current location, thereby improving the rescue efficiency. In addition, determining pedestrian behavior based on water ripple patterns can further improve the accuracy of pedestrian behavior recognition in rivers.

[0090] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0091] Next, in the embodiment of the present disclosure, a device for identifying pedestrian behavior in a river is also provided. Figure 3As shown in , the river pedestrian behavior recognition device 300 can be composed of a video data acquisition module 301, a pedestrian trajectory determination module 302, a weight score determination module 303, a feature fusion module 304 and a behavior classification module 305. Among them: the video data acquisition module can be used to acquire video stream data in the target river scene; the pedestrian trajectory determination module can be used to perform pedestrian detection and pedestrian tracking on the video stream data, and determine the pedestrian trajectory image sequence corresponding to the target pedestrian; the weight score determination module can be used to determine the weight scores corresponding to different images in the pedestrian trajectory image sequence, and determine the multi-channel image sequence corresponding to the target pedestrian according to the weight score; the feature fusion module can be used to convert the multi-channel image sequence into a grayscale image sequence, and perform feature fusion on the multi-channel image sequence and the grayscale image sequence to obtain fusion features; the behavior classification module can be used to input the fusion features into a pre-trained visual language model to obtain the behavior classification corresponding to the target pedestrian.

[0092] It should be noted that the specific details of each part of the above-mentioned device for identifying pedestrian behavior in a river have been described in detail in the implementation method of the method for identifying pedestrian behavior in a river. The undisclosed details can be found in the implementation method of the method part, and will not be repeated here.

[0093] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above-mentioned method for identifying the behavior of pedestrians in a river is also provided.

[0094] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0095] Refer to the following Figure 4 hereinafter describes an electronic device 400 according to such an embodiment of the present disclosure. Figure 4 The electronic device 400 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0096] like Figure 4 As shown, the electronic device 400 is in the form of a general computing device. The components of the electronic device 400 may include but are not limited to: the at least one processing unit 410, the at least one storage unit 420, a bus 430 connecting different system components (including the storage unit 420 and the processing unit 410), and a display unit 440.

[0097] The storage unit stores program codes, which can be executed by the processing unit 410, so that the processing unit 410 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0098] The storage unit 420 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 421 and / or a cache memory unit 422 , and may further include a read-only memory unit (ROM) 423 .

[0099] The storage unit 420 may also include a program / utility 424 having a set (at least one) of program modules 425, such program modules 425 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0100] Bus 430 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0101] The electronic device 400 may also communicate with one or more external devices 470 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 400, and / or may communicate with any device that enables the electronic device 400 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 450. Furthermore, the electronic device 400 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 460. As shown, the network adapter 460 communicates with other modules of the electronic device 400 via a bus 430. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0102] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0103] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0104] refer to Figure 5 As shown, a program product 500 for implementing the above-mentioned method for identifying pedestrian behavior in a river according to an embodiment of the present disclosure is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus, or a device.

[0105] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0106] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0107] The program code contained on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, electromagnetic waves, etc., or any suitable combination of the foregoing.

[0108] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0109] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0110] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0111] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0112] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for identifying pedestrian behavior in a river, characterized in that: include: Obtain video stream data in the target river scene; Performing pedestrian detection and pedestrian tracking on the video stream data to determine a pedestrian trajectory image sequence corresponding to a target pedestrian; Determine weight scores corresponding to different images in the pedestrian trajectory image sequence, and determine a multi-channel image sequence corresponding to the target pedestrian according to the weight scores; Converting the multi-channel image sequence into a grayscale image sequence, and performing feature extraction and feature fusion on the multi-channel image sequence and the grayscale image sequence to obtain fusion features; The fused features are input into a pre-trained visual language model to obtain a behavior classification corresponding to the target pedestrian.

2. The method for identifying pedestrian behavior in a river according to claim 1, characterized in that: The performing pedestrian detection and pedestrian tracking on the video stream data to determine a pedestrian trajectory image sequence corresponding to the target pedestrian includes: Preprocessing the images in the video stream data to obtain preprocessed video stream data; Performing pedestrian detection on the preprocessed video stream data to obtain a first target detection frame of the target pedestrian in the current image; Use the Kalman filter to predict the position of the target pedestrian in the next frame of the image and obtain the predicted detection frame; Matching a second target detection frame of the target pedestrian in the next frame of image with the predicted detection frame, and determining a corresponding relationship between the second target detection frame and the target pedestrian according to the matching result; The pedestrian detection and matching process are repeated to obtain a pedestrian trajectory image sequence corresponding to the target pedestrian.

3. The method for identifying pedestrian behavior in a river according to claim 2, characterized in that: The step of repeating the pedestrian detection and matching process to obtain a pedestrian trajectory image sequence corresponding to the target pedestrian includes: Repeating the pedestrian detection and matching process to determine a set of detection frames corresponding to the target pedestrian; Performing non-maximum suppression processing on the detection frame set to obtain a target detection frame group; A pedestrian trajectory image sequence corresponding to the target pedestrian is determined based on the target detection frame group.

4. The method for identifying pedestrian behavior in a river according to claim 1, characterized in that: The step of determining weight scores corresponding to different images in the pedestrian trajectory image sequence, and determining a multi-channel image sequence corresponding to the target pedestrian according to the weight scores, comprises: Calculating pixel differences between a current image and adjacent frame images in the pedestrian trajectory image sequence to obtain pixel difference values; The weight score is determined according to the pixel difference value, and the multi-channel image sequence is determined according to the image whose weight score is not less than a weight threshold.

5. The method for identifying pedestrian behavior in a river according to claim 1, characterized in that: The extracting and fusing features of the multi-channel image sequence and the grayscale image sequence to obtain fused features includes: Using a convolutional neural network to extract features from the multi-channel image sequence to obtain first image features, where the first image features include one or more of color features, texture features, and light intensity features; Extracting features from the grayscale image sequence based on a local feature description operator to obtain second image features, where the second image features include one or more of structural features, edge features, and corner features; splicing the first image feature and the second image feature to obtain a spliced ​​feature; The feature weights corresponding to different channels in the spliced ​​features are determined through an attention module, and the features corresponding to different channels are weighted combined based on the feature weights to obtain the fused features.

6. The method for identifying pedestrian behavior in a river according to claim 1, characterized in that: Also includes: In response to the behavior corresponding to the target pedestrian being classified as falling into water, acquiring an environmental image of a current location of the target pedestrian; A target rescue path is determined according to the road information in the environment image.

7. The method for identifying pedestrian behavior in a river according to claim 1, characterized in that: Also includes: In response to the result of the pedestrian detection being that the target pedestrian is located in the river, acquiring a river image at the location of the target pedestrian; The behavior classification corresponding to the target pedestrian is determined according to the water ripple pattern in the river image.

8. A device for identifying pedestrian behavior in a river, characterized in that: include: A video data acquisition module is used to acquire video stream data in a target river scene; A pedestrian trajectory determination module is used to perform pedestrian detection and pedestrian tracking on the video stream data to determine a pedestrian trajectory image sequence corresponding to a target pedestrian; A weight score determination module, used to determine the weight scores corresponding to different images in the pedestrian trajectory image sequence, and determine the multi-channel image sequence corresponding to the target pedestrian according to the weight scores; A feature fusion module, used for converting the multi-channel image sequence into a grayscale image sequence, and performing feature extraction and feature fusion on the multi-channel image sequence and the grayscale image sequence to obtain fusion features; The behavior classification module is used to input the fusion features into a pre-trained visual language model to obtain the behavior classification corresponding to the target pedestrian.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the method for identifying pedestrian behavior in a river as described in any one of claims 1-7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying pedestrian behavior in a river channel as described in any one of claims 1 to 7 is implemented.

Images