Pedestrian trajectory analysis method, system, image acquisition device, equipment and medium

Abstract

The present invention provides a pedestrian trajectory analysis method, system, image acquisition device, equipment, and medium. The analysis method includes: acquiring a frame image; inputting the frame image into a pedestrian trajectory analysis model to obtain a pedestrian trajectory analysis result; wherein the pedestrian trajectory analysis result includes a marked image, which is an image obtained by marking each pedestrian in the frame image with a detection box; the pedestrian trajectory analysis model is obtained by inputting a sample image including pedestrians marked with detection boxes into an improved YOLO model for training. The present invention can realize pedestrian data collection in different perspective scenes in a community outdoor road environment, automatically detect pedestrians from video images through a deep learning method, draw detection boxes for pedestrians to mark them, and their activity trajectories, and can investigate the patterns of community population mobility. It provides effective equipment and method support for community pedestrian data collection and pattern analysis in multi-perspective scenarios, with high detection accuracy and ease of implementation.

Description

Technical Field

[0001] The present invention relates to the field of surveying and mapping geographic information technology, and in particular to a pedestrian trajectory analysis method, system, image acquisition device, equipment and medium. Background Art

[0002] In contemporary society, people's main daily activities take place in community settings. Many objective factors affect the activities of pedestrians within different communities, such as weed-covered roads where few people walk. In other words, the activities of pedestrians within a community also reflect the community's internal information from another perspective. For example, pedestrians can often be seen gathering in community parks, and popular community cafes increase the number of pedestrians visiting the community. Modern community planning places greater emphasis on the humanity and livability of the community, which brings new challenges to the renovation and upgrading of community public infrastructure. To solve the above problems, it is necessary to rationally analyze the activities of pedestrians within the community and further summarize the patterns of pedestrian activities.

[0003] Currently, the analysis methods for pedestrian activity patterns can be divided into three categories, which can be summarized as follows:

[0004] (1) Based on the traditional survey and analysis method, the survey and evaluation of pedestrian activities is conducted from a new perspective, using partial data from a full survey to generalize to all objects and study general rules.

[0005] (2) The analysis method based on location data and traffic data is to further analyze the statistics of location and number of pedestrians;

[0006] (3) Based on the crowd gathering and simulation analysis method, taking into account the spatial characteristics of pedestrian activities, the pedestrian gathering situation is analyzed by constructing a reasonable model, and a certain amount of simulation experiments are carried out under conditions as close to reality as possible.

[0007] For the first approach, questionnaires generally fail to adequately summarize the activity status of all pedestrians in an area. The limited statistical categories and scenarios hinder its development. For the second approach, while location information and pedestrian flow statistics can reflect the specific distribution of pedestrians at specific locations within a scene, the short temporal dimension of the statistics makes long-term real-time monitoring difficult. For the third approach, crowd aggregation models primarily describe pedestrian distribution in spatial terms, and simulations are often based on idealized model parameters. This results in poor simulation results for real-world scenarios, with significant deviations from actual conditions.

[0008] In summary, considering the complex environment of community outdoor streets, it is difficult for existing technologies to collect and analyze data in different street environments and with different numbers of pedestrian targets. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to overcome the defect in the prior art that data collection and analysis for different street environments and different numbers of pedestrian targets are relatively difficult, and to provide a pedestrian trajectory analysis method, system, image acquisition device, equipment and medium.

[0010] The present invention solves the above technical problems through the following technical solutions:

[0011] The present invention provides a pedestrian trajectory analysis method, comprising:

[0012] Get frame image;

[0013] Inputting the frame image into a pedestrian trajectory analysis model to obtain a pedestrian trajectory analysis result;

[0014] Among them, the pedestrian trajectory analysis result includes a marked image, which is an image after each pedestrian in the frame image is marked with a detection box. The pedestrian trajectory analysis model is obtained by training the improved YOLO (an object detection model) model by inputting a sample image including pedestrians marked with a detection box.

[0015] Preferably, the improved YOLO model replaces the feature extraction network of the existing YOLO model with a selected layer of a deep learning neural network model.

[0016] Preferably, the neural network model includes at least one of a ResNet-50 (a neural network model) model, a MobileNetv2 (a neural network model) model and a ResNet-18 (a neural network model) model.

[0017] Preferably, when the neural network model includes the ResNet-50 model, the improved YOLO model is divided into two types: ResS-YOLO model and ResB-YOLO model;

[0018] In the ResB-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model;

[0019] In the ResS-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model, and the 32nd to 40th layers of the ResNet-50 model are deleted.

[0020] Preferably, the pedestrian trajectory analysis model includes a first analysis model and a second analysis model;

[0021] The first analysis model is obtained by training the ResS-YOLO model, and the second analysis model is obtained by training the ResB-YOLO model;

[0022] The pedestrian trajectory analysis result also includes a trajectory line of each pedestrian, where the trajectory line is a line connecting the center points of the detection boxes corresponding to the same pedestrian in chronological order;

[0023] The step of inputting the frame image into the pedestrian trajectory analysis model to obtain the pedestrian trajectory analysis result comprises:

[0024] Acquiring a transmission rate of the frame image;

[0025] When the transmission rate is less than a rate threshold, inputting the frame image into the first analysis model to obtain the marking image;

[0026] When the transmission rate is greater than or equal to the rate threshold, the frame image is input into the second analysis model to obtain the marking image and the trajectory line.

[0027] Preferably, during the training process of the pedestrian trajectory analysis model, the loss function Loss is expressed by the following formula:

[0028]

[0029] S 2 =S w *S h ;

[0030]

[0031] Among them, N T Indicates the number of positive samples, N F represents the number of negative samples, λ represents the weight coefficient, i represents the grid number, j represents the corresponding detection unit number in the i-th grid, x and y represent the predicted horizontal and vertical coordinates of the center point of the detection box, respectively. and They represent the actual horizontal and vertical coordinates of the center point, w represents the predicted width of the detection box, and h represents the predicted height of the detection box. and Respectively represent the actual width and height of the detection box, B represents the number of the detection boxes, S 2 Indicates the total number of grids, S w Indicates the number of grids per row, S hRepresents the number of grids in each column, C represents the confidence of the prediction of the detection box, Represents the actual confidence of the detection box, P represents the predicted object category probability, Represents the actual object category probability, Pr(Object) represents the probability of the target object existing in the grid cell, Indicates the overlap rate between the detection box predicted by the model and the actual object;

[0032] When the detection unit is a positive sample, is 1, is 0; when the detection unit is a negative sample, is 0, is 1.

[0033] Preferably, the step of acquiring the frame image includes:

[0034] Acquire multiple original frame images captured by multiple cameras at the same time point; wherein the shooting ranges of two adjacent cameras overlap or connect;

[0035] The multiple original frame images at the same time point are stitched together to obtain a panoramic image, and the panoramic image is used as the frame image.

[0036] The present invention also provides a pedestrian trajectory analysis system, comprising: an image acquisition module and an analysis result generation module;

[0037] The image acquisition module is used to acquire frame images;

[0038] The analysis result generating module is used to input the frame image into the pedestrian trajectory analysis model to obtain the pedestrian trajectory analysis result;

[0039] Among them, the pedestrian trajectory analysis result includes a marked image, which is an image after each pedestrian in the frame image is marked with a detection box. The pedestrian trajectory analysis model is obtained by training the improved YOLO model by inputting sample images including pedestrians marked with detection boxes.

[0040] Preferably, the improved YOLO model replaces the feature extraction network of the existing YOLO model with a selected layer of a deep learning neural network model.

[0041] Preferably, the neural network model includes at least one of a ResNet-50 model, a MobileNet v2 model and a ResNet-18 model.

[0042] Preferably, when the neural network model includes the ResNet-50 model, the improved YOLO model is divided into two types: ResS-YOLO model and ResB-YOLO model;

[0043] In the ResB-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model;

[0044] In the ResS-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model, and the 32nd to 40th layers of the ResNet-50 model are deleted.

[0045] Preferably, the pedestrian trajectory analysis model includes a first analysis model and a second analysis model;

[0046] The first analysis model is obtained by training the ResS-YOLO model, and the second analysis model is obtained by training the ResB-YOLO model;

[0047] The pedestrian trajectory analysis result also includes a trajectory line of each pedestrian, where the trajectory line is a line connecting the center points of the detection boxes corresponding to the same pedestrian in chronological order;

[0048] The analysis result generating module includes: a transmission rate acquiring unit, a first analysis generating unit and a second analysis generating unit;

[0049] The transmission rate acquisition unit is used to acquire the transmission rate of the frame image;

[0050] The first analysis and generation unit is configured to input the frame image into the first analysis model to obtain the marking image when the transmission rate is less than a rate threshold;

[0051] The second analysis generation unit is configured to input the frame image into the second analysis model to obtain the marking image and the trajectory line when the transmission rate is greater than or equal to the rate threshold.

[0052] Preferably, during the training process of the pedestrian trajectory analysis model, the loss function Loss is expressed by the following formula:

[0053]

[0054]

[0055] S 2 =S w *S h ;

[0056]

[0057] Among them, N T Indicates the number of positive samples, N F represents the number of negative samples, λ represents the weight coefficient, i represents the grid number, j represents the corresponding detection unit number in the i-th grid, x and y represent the predicted horizontal and vertical coordinates of the center point of the detection box, respectively. and They represent the actual horizontal and vertical coordinates of the center point, w represents the predicted width of the detection box, and h represents the predicted height of the detection box. and Respectively represent the actual width and height of the detection box, B represents the number of the detection boxes, S 2 Indicates the total number of grids, S w Indicates the number of grids per row, S h Represents the number of grids in each column, C represents the confidence of the prediction of the detection box, Represents the actual confidence of the detection box, P represents the predicted object category probability, Represents the actual object category probability, Pr(Object) represents the probability of the target object existing in the grid cell, Indicates the overlap rate between the detection box predicted by the model and the actual object;

[0058] When the detection unit is a positive sample, is 1, is 0; when the detection unit is a negative sample, is 0, is 1.

[0059] Preferably, the image acquisition module includes: an original frame image acquisition unit and a splicing unit;

[0060] The original frame image acquisition unit is used to acquire multiple original frame images captured by multiple cameras at the same time point; wherein the shooting ranges of two adjacent cameras overlap or connect;

[0061] The stitching unit is used to stitch the multiple original frame images at the same time point to obtain a panoramic image, and use the panoramic image as the frame image.

[0062] The present invention also provides an image acquisition device, comprising: a controller, a plurality of cameras, a moving part corresponding to the cameras, and a bracket;

[0063] The bracket is provided with a slide rail corresponding to the camera, and the slide rail surrounds the central axis of the image acquisition device;

[0064] Each of the moving parts includes: a rotating motor and a connecting rod;

[0065] The controller is electrically connected to each of the rotating motors and each of the cameras respectively;

[0066] The first end of the connecting rod is fixedly connected to the corresponding camera, and the second end of the connecting rod is transmission-connected to the corresponding rotating motor;

[0067] The rotary motor is used to drive the first end of the corresponding connecting rod to rotate along the corresponding slide rail;

[0068] The controller is used to control the rotating motor to drive;

[0069] The controller is also used to receive the frame image captured by the camera.

[0070] Preferably, the controller is further configured to obtain a pedestrian trajectory analysis result of the current frame image within a preset time period; wherein the pedestrian trajectory analysis result is obtained using the aforementioned pedestrian trajectory analysis method;

[0071] The controller is also used to obtain pedestrian distribution data based on the pedestrian trajectory analysis results, and obtain a target scheduling method based on the distribution data and the matching relationship between the distribution data and the scheduling method, and control the rotating motor to drive according to the target scheduling method so that the distribution of the camera conforms to the target scheduling method; wherein, the scheduling method includes the rotation direction and rotation angle of the camera.

[0072] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned pedestrian trajectory analysis method when executing the computer program.

[0073] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the aforementioned pedestrian trajectory analysis method when executed by a processor.

[0074] The positive progress of the present invention is that it can realize the pedestrian data collection of different perspective scenes in the outdoor road environment of the community, automatically detect pedestrians from video image data through deep learning methods, and draw pedestrian detection boxes to mark the pedestrians and their activity trajectories. It can investigate the mobility patterns of the community population, and provide effective equipment and method support for the community pedestrian data collection and pattern analysis in multi-perspective scenes. At the same time, it provides a simpler design principle for multi-camera time synchronization, and the detection accuracy of pedestrians and their activity trajectories is high and easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is a flowchart of the pedestrian trajectory analysis method according to Example 1 of the present invention.

[0076] Figure 2 This is a flow chart of the ResS-YOLO model in the pedestrian trajectory analysis method of Example 1 of the present invention.

[0077] Figure 3 This is a flow chart of the ResB-YOLO model in the pedestrian trajectory analysis method of Example 1 of the present invention.

[0078] Figure 4 This is a structural diagram of the detection network of the existing YOLO model in the pedestrian trajectory analysis method of Example 1 of the present invention.

[0079] Figure 5 This is a flowchart of a specific implementation of step S12 in the pedestrian trajectory analysis method of embodiment 1 of the present invention.

[0080] Figure 6 This is a flowchart of a specific implementation of step S11 in the pedestrian trajectory analysis method of embodiment 1 of the present invention.

[0081] Figure 7 Schematic diagram of the modules of the pedestrian trajectory analysis system according to embodiment 2 of the present invention.

[0082] Figure 8 This is a front view structural diagram of the image acquisition device of Example 3 of the present invention.

[0083] Figure 9 This is a left-view structural diagram of the image acquisition device of Example 3 of the present invention.

[0084] Figure 10 This is a top view of the structure of the image acquisition device of Example 3 of the present invention.

[0085] Figure 11 Schematic diagram of the clock synchronization principle in the image acquisition device of Example 3 of the present invention.

[0086] Figure 12 Schematic diagram of the standard layout of the image acquisition device in embodiment 3 of the present invention.

[0087] Figure 13 Schematic diagram of the I-type layout in the image acquisition device of Example 3 of the present invention.

[0088] Figure 14 Schematic diagram of type II layout in the image acquisition device of embodiment 3 of the present invention.

[0089] Figure 15Schematic diagram of the type III layout in the image acquisition device of embodiment 3 of the present invention.

[0090] Figure 16 Schematic diagram of the IV-type layout in the image acquisition device of Example 3 of the present invention.

[0091] Figure 17 This is an example of the scheduling results in the image acquisition device of embodiment 3 of the present invention.

[0092] Figure 18 This is a schematic structural diagram of an electronic device according to embodiment 4 of the present invention. DETAILED DESCRIPTION

[0093] The present invention is further described below by way of examples, but the present invention is not limited to the scope of the examples.

[0094] Example 1

[0095] This embodiment provides a method for analyzing pedestrian trajectories. Figure 1 , the analysis methods include:

[0096] S11. Acquire a frame image.

[0097] S12. Input the frame image into the pedestrian trajectory analysis model to obtain a pedestrian trajectory analysis result, wherein the pedestrian trajectory analysis result includes a marked image, which is an image obtained by marking each pedestrian in the frame image with a detection box. The pedestrian trajectory analysis model is obtained by inputting sample images including pedestrians marked with detection boxes into an improved YOLO model for training.

[0098] Among them, the reason for using the YOLO model for improvement is that the YOLO model has the following advantages:

[0099] (1) There is no need to execute complex processes to deal with regression problems. The results can be quickly obtained by running the algorithm directly on the image to be detected.

[0100] (2) The YOLO model uses the full image as the viewing angle when performing target detection. It does not use a sliding window. The target and surrounding neighborhood information are used for feature encoding to improve detection accuracy.

[0101] (3) The YOLO model has little ability to decompose data of non-detection categories and is more robust.

[0102] The improved YOLO model is a deep learning model. The pedestrian trajectory analysis model trained by this model implements a deep learning method. It can automatically detect pedestrians from video image data, draw detection boxes for pedestrians, and quickly and efficiently identify pedestrians in complex environment scenes.

[0103] In this embodiment, pedestrian data can be collected in different perspective scenes in the community outdoor road environment. Pedestrians can be automatically detected from video image data through deep learning methods, and detection boxes for pedestrians can be drawn to mark them. The community population mobility patterns can be investigated, and an effective method support is provided for community pedestrian data collection and pattern analysis in multi-perspective scenes. The detection accuracy of pedestrians and their activity trajectories is high and easy to implement.

[0104] In specific implementation, the improved YOLO model replaces the feature extraction network of the existing YOLO model with selected layers of the deep learning neural network model.

[0105] The selected layer is determined based on the actual needs of feature extraction and the matching of data channels. The improved YOLO model can not only ensure the better performance of the feature extraction network, but also retain the advantages of the existing YOLO model.

[0106] Preferably, the existing YOLO model adopts the YOLO v2 model.

[0107] The original feature extraction network in the YOLO v2 model is the DarkNet-19 model (a feature extraction network). Here, it is completely replaced with a deep learning neural network model, which can output image data after 16 times downsampling to meet the needs of pedestrian recognition on complex community roads.

[0108] In a specific implementation, the neural network model includes at least one of a ResNet-50 model, a MobileNet v2 model, and a ResNet-18 model.

[0109] Among them, a suitable network model can be selected as the feature extraction network according to the application scenario and the complexity of the problem.

[0110] Preferably, the neural network model uses a ResNet-50 model. The ResNet-50 model has better feature extraction effect and is more suitable for the improved YOLO model.

[0111] In specific implementation, when the neural network model includes the ResNet-50 model, the improved YOLO model is divided into two types: the ResS-YOLO model and the ResB-YOLO model.

[0112] In the ResB-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model.

[0113] In the ResS-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model, and the 32-40 layers of the ResNet-50 model are deleted.

[0114] In the ResB-YOLO model, the selected layers of the ResNet-50 model are layers 1 to 49. Due to data channel matching requirements, the 49th layer of the ResNet-50 model is suitable for connecting to the detection network of the existing YOLO model. You can choose to connect one of the 49th layers to the input of the detection network as needed.

[0115] In the ResS-YOLO model, the selected layers of the ResNet-50 model are layers 1-31 and layers 41-49.

[0116] Specifically, the selected layers of the ResNet-50 model are divided into five sub-layers: STAGE 0-4.

[0117] STAGE 1-4 all use two convolutional structures, BTNK1 and BTNK2. BTNK1 undergoes four convolution (CONV) operations and one RELU (rectified linear unit) operation, while BTNK2 undergoes three convolution (CONV) operations and one RELU operation.

[0118] Figure 2 The process structure of the ResS-YOLO model is shown. Figure 3 The process structure of the ResB-YOLO model is shown. Figure 4 The figure shows the structure of the detection network of the existing YOLO module. This part is the detailed structure of the YOLO Module in STAGE 4.

[0119] BTNK1 represents the case where the number of input and output channels is different, and BTNK2 represents the case where the number of input and output channels is the same. C and C1 represent the number of channels, W represents the width and height, and S represents the stride.

[0120] BTNK1 has four variable parameters: C, W, C1, and S. Compared to BTNK2, BTNK1 has an additional convolutional layer on the right side, which is represented by the function G(x). BTNK1 accommodates the case where the input x and the output F(x) have different numbers of channels. This added convolutional layer transforms x into G(x), matching the difference in input and output dimensions (G(x) and F(x) have the same number of channels), allowing the sum F(x) + G(x). In summary, the output of BTNK1 is expressed as: H(x) = F(x) + G(x).

[0121] BTNK2 has two variable parameters, C and W, which are the C and W in the input shape (C, W, W). Let the input shape be (C, W, W) be x, and let the three convolution blocks on the left side of BTNK2 (and the related BN and RELU) be the function F(x). After adding the two together (F(x) + x) and then passing through a ReLU activation function, we get the output of BTNK2. The output shape is still (C, W, W), which is the case mentioned above when BTNK2 corresponds to the case where the input x and the output F(x) have the same number of channels.

[0122] In summary, the output expression of BTNK2 is: H(x)=F(x)+x.

[0123] The RELU activation function uses the following formula:

[0124]

[0125] Regarding the convolution operation used here, assume that the input image size is W*W, the convolution kernel size is F*F, the step size is S, the padding width is P, and the output image size after convolution is N*N:

[0126] N = (W-F+2P) / S+1.

[0127] According to the above rules, in the first convolution operation of BTNK1, the input image size is W*W, the convolution kernel size is 1*1, the step size is S, the number of channels is C1, the padding width is 0.5, and the output size is N1=(W-1+0.5*2) / S+1=W / S+1; in the second convolution operation, the image size is N1*N1, the convolution kernel size is 3*3, the step size is 1, the padding width is 0, the number of channels is C1, and the output size is N2=(N1-3) / 1+1=W / S+1-3+1=W / S-1; in the third convolution operation, the image size is N2*N2, The convolution kernel size is 1*1, the step size is 1, the padding width is 0, the number of channels is C1*4, and the output size is N3=(N2-1) / 1+1=W / S-1-1+1=W / S-1; the image size of the fourth convolution operation is W*W, the convolution kernel size is 1*1, the step size is 1, the padding width is 0.5, the number of channels is C1*4, and the output size is N4*N4, N4=W / S+1; the number of channels of the third and fourth convolutions is the same, both are C1*4. After one RELU operation, they are merged to get N*N, N=(N3+N4) / 2=W / S.

[0128] Similarly, in the first convolution operation of BTNK2, the input image size is W*W, the convolution kernel size is 1*1, the step size is S, the number of channels is C / 4, the padding width is 1, and the output size is N1=(W-1+2) / 1+1=W+2; in the second convolution operation, the image size is N1*N1, the convolution kernel size is 3*3, the step size is 1, the padding width is 0, the number of channels is C1 / 4, and the output size is N2=(N1-3) / 1+1=W+2-3+1=W; in the third convolution operation, the image size is N2*N2, the convolution kernel size is 1*1, the step size is 1, the padding width is 0, the number of channels is C1, and the output size is N3=(N2-1) / 1+1=W-1+1=W; the third convolution is consistent with the initial number of channels, both C1. After one RELU operation, the merged result is N*N, N=W.

[0129] In a specific implementation, the pedestrian trajectory analysis model includes a first analysis model and a second analysis model.

[0130] The first analysis model is trained by the ResS-YOLO model, and the second analysis model is trained by the ResB-YOLO model.

[0131] The pedestrian trajectory analysis results also include the trajectory line of each pedestrian, which is a line connecting the center points of the detection boxes corresponding to the same pedestrian in chronological order.

[0132] Reference Figure 5 , step S12 includes:

[0133] S121. Obtain the transmission rate of the frame image.

[0134] S122: When the transmission rate is less than the rate threshold, input the frame image into the first analysis model to obtain a marked image.

[0135] S123 : When the transmission rate is greater than or equal to the rate threshold, input the frame image into the second analysis model to obtain a marked image and a trajectory line.

[0136] Among them, the current application scenario can be determined according to the transmission rate of the frame image. If the transmission rate is less than the rate threshold, the current application scenario is a small data scenario; if the transmission rate is greater than or equal to the rate threshold, the current application scenario is a big data scenario.

[0137] The data processing capability of the second analysis model is higher than that of the first analysis model.

[0138] The rate threshold may be set according to actual needs, for example, to 800 Kbps (kilobits per second).

[0139] In this embodiment, for continuously detected pedestrian targets, the center point is taken in the detection box between adjacent frames, and lines are connected to draw the trajectory line of the pedestrian. The trajectory line can reflect the activity trajectory of the pedestrian. The detection box and the trajectory line can be used to investigate the mobility patterns of the community population, further providing an effective method support for community pedestrian data collection and pattern analysis in multi-perspective scenarios. The detection accuracy of pedestrians and their activity trajectories is high and easy to implement.

[0140] In specific implementation, during the training process of the pedestrian trajectory analysis model, the loss function Loss is expressed using the following formula:

[0141]

[0142] S 2 =S w *S h .

[0143]

[0144] Among them, N T Indicates the number of positive samples, N F Represents the number of negative samples, λ represents the weight coefficient, i represents the grid number, j represents the corresponding detection unit number in the i-th grid, x and y represent the predicted horizontal and vertical coordinates of the center point of the detection box, respectively. and Represent the actual horizontal and vertical coordinates of the center point, w represents the predicted width of the detection box, and h represents the predicted height of the detection box. and Respectively represent the actual width and height of the detection box, B represents the number of detection boxes, S 2 Indicates the total number of grids, S w Indicates the number of grids per row, S h Represents the number of grids in each column, C represents the confidence of the prediction of the detection box, Represents the actual confidence of the detection box, P represents the predicted object category probability, Represents the actual object category probability, Pr(Object) represents the probability of the target object existing in the grid cell, Indicates the overlap rate between the detection box predicted by the model and the actual object.

[0145] When the detection unit is a positive sample, is 1, is 0; when the detection unit is a negative sample, is 0, is 1.

[0146] The derivation process of the loss function is as follows:

[0147] The loss function mainly consists of four variables, namely loss xy 、loss wh 、loss conf and loss class , corresponding to four loss function terms, the initial loss function constructed is:

[0148]

[0149] S 2 =S w *S h .

[0150]

[0151] Among them, loss xy Represents the loss function term corresponding to the coordinates of the center position of the detection box, loss wh Indicates the loss function term corresponding to the size of the detection box, loss conf Represents the loss function term corresponding to the confidence level, loss class Represents the loss function term corresponding to the type of the detected object, N T Indicates the number of positive samples, N F Represents the number of negative samples, λ represents the weight coefficient, i represents the grid number, j represents the corresponding detection unit number in the i-th grid, x and y represent the predicted horizontal and vertical coordinates of the center point of the detection box, respectively. and Represent the actual horizontal and vertical coordinates of the center point, w represents the predicted width of the detection box, and h represents the predicted height of the detection box. and Respectively represent the actual width and height of the detection box, B represents the number of detection boxes, S 2 Indicates the total number of grids, S w Indicates the number of grids per row, S h Represents the number of grids in each column, C represents the confidence of the prediction of the detection box, Represents the actual confidence of the detection box, P represents the predicted object category probability, Represents the actual object category probability, Pr(Object) represents the probability of the target object existing in the grid cell, Indicates the overlap rate between the detection box predicted by the model and the actual object.

[0152] When the detection unit is a positive sample, is 1, is 0; when the detection unit is a negative sample, is 0, is 1.

[0153] The YOLO model algorithm essentially grids the entire image data and then performs a single detection. This ensures that the number of positive samples in a detection unit is either 0 or 1, while the number of negative samples is much greater than 1. This results in a significant disparity in the number of positive and negative samples in the entire image data, with the difference reaching orders of magnitude. This imbalance between positive and negative samples reduces the efficiency of algorithm model training. Furthermore, a large number of negative samples can mask positive samples, leading to model degradation after training. To address this issue, the existing YOLO model uses a weight scaling coefficient to suppress the impact of negative samples on the loss function and increase the contribution of positive samples, thereby improving learning efficiency. However, simply assigning a value to the weight scaling coefficient can only be controlled within a certain threshold; otherwise, overfitting or training errors can occur.

[0154] Based on the initial loss function formula, it is found that loss conf Is the core that needs to be optimized, loss conf The following formula can be used:

[0155]

[0156] Taking the derivative of the above formula, we can get:

[0157]

[0158] Through the above formula, it is found that the derivative of confidence is positively correlated with the difference between the predicted value and the true value, so:

[0159]

[0160] Among them, ΔW represents the increment of the loss function to the variable, η W Represents the learning rate.

[0161] Based on the above three equations, we found that if the difference in the number of positive and negative samples is too large, the gradient features will be unclear, ultimately leading to detection failure. Based on this, we constructed the adaptive adjustment coefficient shown below to balance the impact of this difference.

[0162]

[0163] The adaptive adjustment coefficient can be generated adaptively according to the difference between the predicted value and the true value, where γ represents the base adjustment factor and b represents the exponential adjustment factor. b effectively controls the size of the adjustment coefficient by controlling the size of the exponential part, and effectively controls the size of α by using the high gradient change principle of the exponential function. When the sample under test is a positive sample is 1 when the sample is a negative sample is 0. After introducing the new adaptive scale factor, we have:

[0164]

[0165] in, Represents the confidence loss function term.

[0166] Taking the derivative of the above formula, we have:

[0167]

[0168] Therefore, after adding the adaptive scaling coefficient, the greater the difference between the predicted value and the true value, the greater the derivative of the loss term will be, and vice versa.

[0169] It can be found that when the value of b is too large, the gradient of positive and negative samples will change too much, while when the value of b is too small, the gradient of positive and negative samples will be insensitive to the change. When γ is 0, the loss function is consistent with the traditional loss function. Finally, after testing, the parameters are selected when b = 2 and γ = 2, which can be used to derive the formula for the loss function mentioned above.

[0170] In specific implementation, during the model training process, the detection accuracy of the detected pedestrian targets is expressed by the detection box accuracy and trajectory drawing accuracy.

[0171] The relevant indicators of detection accuracy are as follows.

[0172] Precision pp: This indicator indicates the accuracy of detecting individual pedestrian targets.

[0173] Recall rate r: This metric indicates how accurately the number of pedestrians is identified.

[0174] Accuracy a: This indicator indicates the accuracy of distinguishing pedestrians and other ground objects.

[0175] Among them, TP represents the number of positive samples predicted as positive samples, FP is the number of negative samples predicted as positive samples, FN is the number of positive samples that are mistakenly considered to be negative samples, and TN is the number of negative samples that are correctly considered to be negative samples.

[0176] The precision, recall and accuracy can be statistically plotted.

[0177] When the precision is greater than the precision threshold (e.g., 75%), the recall is greater than the recall threshold (e.g., 80%), and the accuracy is greater than the accuracy threshold (e.g., 85%), the model is trained and can be used.

[0178] You can set the precision threshold, recall threshold, and accuracy threshold according to actual needs.

[0179] When implementing it, refer to Figure 6 , step S11 includes:

[0180] S111: Acquire multiple original frame images captured by multiple cameras at the same time point, wherein the shooting ranges of two adjacent cameras overlap or connect.

[0181] S112 , stitching together multiple original frame images at the same time point to obtain a panoramic image, and using the panoramic image as the frame image.

[0182] In this embodiment, the original frame images captured by multiple cameras can be spliced ​​into a panoramic image, so that the activity trajectory of pedestrians can be analyzed in a larger range.

[0183] Alternatively, instead of splicing the original frame images, the frame images from multiple cameras can be analyzed separately to obtain multiple pedestrian trajectory analysis results. These multiple pedestrian trajectory analysis results are then combined for further processing. For example, after a pedestrian is detected in a certain frame, the pedestrian is numbered. Between frames, the same pedestrian is assigned an ID number based on their characteristics. The same person with the same characteristics has only one unique number. Pedestrians with the same ID number between different frames are considered the same person. The detection box here marks the pedestrian target individual, and the recorded information is the x-coordinate and y-coordinate relative to the upper left corner of the image, as well as the width w and height h of the box.

[0184] Example 2

[0185] This embodiment provides a pedestrian trajectory analysis system, referring to Figure 7 The analysis system includes: an image acquisition module 1 and an analysis result generation module 2.

[0186] The image acquisition module 1 is used to acquire frame images.

[0187] The analysis result generating module 2 is used to input the frame image into the pedestrian trajectory analysis model to obtain the pedestrian trajectory analysis result.

[0188] Among them, the pedestrian trajectory analysis results include marked images, which are images after each pedestrian in the frame image is marked with a detection box. The pedestrian trajectory analysis model is obtained by training the improved YOLO model by inputting sample images including pedestrians marked with detection boxes.

[0189] Among them, the reason for using the YOLO model for improvement is that the YOLO model has the following advantages:

[0190] (1) There is no need to execute complex processes to deal with regression problems. The results can be quickly obtained by running the algorithm directly on the image to be detected.

[0191] (2) The YOLO model uses the full image as the viewing angle when performing target detection. It does not use a sliding window. The target and surrounding neighborhood information are used for feature encoding to improve detection accuracy.

[0192] (3) The YOLO model has little ability to decompose data of non-detection categories and is more robust.

[0193] The improved YOLO model is a deep learning model. The pedestrian trajectory analysis model trained by this model implements a deep learning method. It can automatically detect pedestrians from video image data, draw detection boxes for pedestrians, and quickly and efficiently identify pedestrians in complex environment scenes.

[0194] In this embodiment, pedestrian data can be collected in different perspective scenes in the community outdoor road environment. Pedestrians can be automatically detected from video image data through deep learning methods, and detection boxes for pedestrians can be drawn to mark them. The community population mobility patterns can be investigated, and an effective method support is provided for community pedestrian data collection and pattern analysis in multi-perspective scenes. The detection accuracy of pedestrians and their activity trajectories is high and easy to implement.

[0195] In specific implementation, the improved YOLO model replaces the feature extraction network of the existing YOLO model with selected layers of the deep learning neural network model.

[0196] The selected layer is determined based on the actual needs of feature extraction and the matching of data channels. The improved YOLO model can not only ensure the better performance of the feature extraction network, but also retain the advantages of the existing YOLO model.

[0197] Preferably, the existing YOLO model adopts the YOLO v2 model.

[0198] The original feature extraction network in the YOLO v2 model is the DarkNet-19 model (a feature extraction network). Here, it is completely replaced with a deep learning neural network model, which can output image data after 16 times downsampling to meet the needs of pedestrian recognition on complex community roads.

[0199] In a specific implementation, the neural network model includes at least one of a ResNet-50 model, a MobileNet v2 model, and a ResNet-18 model.

[0200] Among them, a suitable network model can be selected as the feature extraction network according to the application scenario and the complexity of the problem.

[0201] Preferably, the neural network model uses a ResNet-50 model. The ResNet-50 model has better feature extraction effect and is more suitable for the improved YOLO model.

[0202] In specific implementation, when the neural network model includes the ResNet-50 model, the improved YOLO model is divided into two types: the ResS-YOLO model and the ResB-YOLO model.

[0203] In the ResB-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model.

[0204] In the ResS-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model, and the 32-40 layers of the ResNet-50 model are deleted.

[0205] In the ResB-YOLO model, the selected layers of the ResNet-50 model are layers 1 through 49. Due to data channel matching requirements, layer 49 of the ResNet-50 model is suitable for connecting to the detection network of the existing YOLO model. You can select a layer in layer 49 to connect to the input of the detection network as needed.

[0206] In the ResS-YOLO model, the selected layers of the ResNet-50 model are layers 1-31 and layers 41-49.

[0207] Specifically, the selected layers of the ResNet-50 model are divided into five sub-layers: STAGE 0-4.

[0208] STAGE 1-4 all use two convolutional structures, BTNK1 and BTNK2. BTNK1 undergoes four convolution (CONV) operations and one RELU (rectified linear unit) operation, while BTNK2 undergoes three convolution (CONV) operations and one RELU operation.

[0209] Figure 2 The process structure of the ResS-YOLO model is shown. Figure 3 The process structure of the ResB-YOLO model is shown. Figure 4 The figure shows the structure of the detection network of the existing YOLO module. This part is the detailed structure of the YOLO Module in STAGE 4.

[0210] BTNK1 represents the case where the number of input and output channels is different, and BTNK2 represents the case where the number of input and output channels is the same. C and C1 represent the number of channels, W represents the width and height, and S represents the stride.

[0211] BTNK1 has four variable parameters: C, W, C1, and S. Compared to BTNK2, BTNK1 has an additional convolutional layer on the right side, which is represented by the function G(x). BTNK1 accommodates the case where the input x and the output F(x) have different numbers of channels. This added convolutional layer transforms x into G(x), matching the difference in input and output dimensions (G(x) and F(x) have the same number of channels), allowing the sum F(x) + G(x). In summary, the output of BTNK1 is expressed as: H(x) = F(x) + G(x).

[0212] BTNK2 has two variable parameters, C and W, which are the C and W in the input shape (C, W, W). Let the input shape be (C, W, W) be x, and let the three convolution blocks on the left side of BTNK2 (and the related BN and RELU) be the function F(x). After adding the two together (F(x) + x) and then passing through a ReLU activation function, we get the output of BTNK2. The output shape is still (C, W, W), which is the case mentioned above when BTNK2 corresponds to the case where the input x and the output F(x) have the same number of channels.

[0213] In summary, the output expression of BTNK2 is: H(x)=F(x)+x.

[0214] The RELU activation function uses the following formula:

[0215]

[0216] Regarding the convolution operation used here, assume that the input image size is W*W, the convolution kernel size is F*F, the step size is S, the padding width is P, and the output image size after convolution is N*N:

[0217] N = (W-F+2P) / S+1.

[0218] According to the above rules, in the first convolution operation of BTNK1, the input image size is W*W, the convolution kernel size is 1*1, the step size is S, the number of channels is C1, the padding width is 0.5, and the output size is N1=(W-1+0.5*2) / S+1=W / S+1; in the second convolution operation, the image size is N1*N1, the convolution kernel size is 3*3, the step size is 1, the padding width is 0, the number of channels is C1, and the output size is N2=(N1-3) / 1+1=W / S+1-3+1=W / S-1; in the third convolution operation, the image size is N2*N2, The convolution kernel size is 1*1, the step size is 1, the padding width is 0, the number of channels is C1*4, and the output size is N3=(N2-1) / 1+1=W / S-1-1+1=W / S-1; the image size of the fourth convolution operation is W*W, the convolution kernel size is 1*1, the step size is 1, the padding width is 0.5, the number of channels is C1*4, and the output size is N4*N4, N4=W / S+1; the number of channels of the third and fourth convolutions is the same, both are C1*4. After one RELU operation, they are merged to get N*N, N=(N3+N4) / 2=W / S.

[0219] Similarly, in the first convolution operation of BTNK2, the input image size is W*W, the convolution kernel size is 1*1, the step size is S, the number of channels is C / 4, the padding width is 1, and the output size is N1=(W-1+2) / 1+1=W+2; in the second convolution operation, the image size is N1*N1, the convolution kernel size is 3*3, the step size is 1, the padding width is 0, the number of channels is C1 / 4, and the output size is N2=(N1-3) / 1+1=W+2-3+1=W; in the third convolution operation, the image size is N2*N2, the convolution kernel size is 1*1, the step size is 1, the padding width is 0, the number of channels is C1, and the output size is N3=(N2-1) / 1+1=W-1+1=W; the third convolution is consistent with the initial number of channels, both C1. After one RELU operation, the merged result is N*N, N=W.

[0220] In a specific implementation, the pedestrian trajectory analysis model includes a first analysis model and a second analysis model.

[0221] The first analysis model is trained by the ResS-YOLO model, and the second analysis model is trained by the ResB-YOLO model.

[0222] The pedestrian trajectory analysis results also include the trajectory line of each pedestrian, which is a line connecting the center points of the detection boxes corresponding to the same pedestrian in chronological order.

[0223] The analysis result generating module 2 includes: a transmission rate acquiring unit 21 , a first analysis generating unit 22 and a second analysis generating unit 23 .

[0224] The transmission rate acquisition unit 21 is used to acquire the transmission rate of the frame image.

[0225] The first analysis generation unit 22 is configured to input the frame image into the first analysis model to obtain a marker image when the transmission rate is less than a rate threshold.

[0226] The second analysis generation unit 23 is configured to input the frame image into the second analysis model to obtain a marked image and a trajectory line when the transmission rate is greater than or equal to the rate threshold.

[0227] Among them, the current application scenario can be determined according to the transmission rate of the frame image. If the transmission rate is less than the rate threshold, the current application scenario is a small data scenario; if the transmission rate is greater than or equal to the rate threshold, the current application scenario is a big data scenario.

[0228] The data processing capability of the second analysis model is higher than that of the first analysis model.

[0229] The rate threshold may be set according to actual needs, for example, to 800 Kbps (kilobits per second).

[0230] In this embodiment, for continuously detected pedestrian targets, the center point is taken in the detection box between adjacent frames, and lines are connected to draw the trajectory line of the pedestrian. The trajectory line can reflect the activity trajectory of the pedestrian. The detection box and the trajectory line can be used to investigate the mobility patterns of the community population, further providing an effective method support for community pedestrian data collection and pattern analysis in multi-perspective scenarios. The detection accuracy of pedestrians and their activity trajectories is high and easy to implement.

[0231] In specific implementation, during the training process of the pedestrian trajectory analysis model, the loss function is expressed as follows:

[0232]

[0233] S 2 =S w *S h .

[0234]

[0235] Among them, N T Indicates the number of positive samples, N F Represents the number of negative samples, λ represents the weight coefficient, i represents the grid number, j represents the corresponding detection unit number in the i-th grid, x and y represent the predicted horizontal and vertical coordinates of the center point of the detection box, respectively. and Represent the actual horizontal and vertical coordinates of the center point, w represents the predicted width of the detection box, and h represents the predicted height of the detection box. and Respectively represent the actual width and height of the detection box, B represents the number of detection boxes, S 2 Indicates the total number of grids, S w Indicates the number of grids per row, S h Represents the number of grids in each column, C represents the confidence of the prediction of the detection box, Represents the actual confidence of the detection box, P represents the predicted object category probability, Represents the actual object category probability, Pr(Object) represents the probability of the target object existing in the grid cell, Indicates the overlap rate between the detection box predicted by the model and the actual object.

[0236] When the detection unit is a positive sample, is 1, is 0; when the detection unit is a negative sample, is 0, is 1.

[0237] The derivation process of the loss function is as follows:

[0238] The loss function mainly consists of four variables, namely loss xy 、loss wh 、loss conf and loss class , corresponding to four loss function terms, the initial loss function constructed is:

[0239]

[0240] S 2 =S w *S h .

[0241]

[0242] Among them, loss xy Represents the loss function term corresponding to the coordinates of the center position of the detection box, loss wh Indicates the loss function term corresponding to the size of the detection box, loss conf Represents the loss function term corresponding to the confidence level, loss class Represents the loss function term corresponding to the type of the detected object, N T Indicates the number of positive samples, N F Represents the number of negative samples, λ represents the weight coefficient, i represents the grid number, j represents the corresponding detection unit number in the i-th grid, x and y represent the predicted horizontal and vertical coordinates of the center point of the detection box, respectively. and Represent the actual horizontal and vertical coordinates of the center point, w represents the predicted width of the detection box, and h represents the predicted height of the detection box. and Respectively represent the actual width and height of the detection box, B represents the number of detection boxes, S 2 Indicates the total number of grids, S w Indicates the number of grids per row, S h Represents the number of grids in each column, C represents the confidence of the prediction of the detection box, Represents the actual confidence of the detection box, P represents the predicted object category probability, Represents the actual object category probability, Pr(Object) represents the probability of the target object existing in the grid cell, Indicates the overlap rate between the detection box predicted by the model and the actual object.

[0243] When the detection unit is a positive sample, is 1, is 0; when the detection unit is a negative sample, is 0, is 1.

[0244] The YOLO model algorithm essentially grids the entire image data and then performs a single detection. This ensures that the number of positive samples in a detection unit is either 0 or 1, while the number of negative samples is much greater than 1. This results in a significant disparity in the number of positive and negative samples in the entire image data, with the difference reaching orders of magnitude. This imbalance between positive and negative samples reduces the efficiency of algorithm model training. Furthermore, a large number of negative samples can mask positive samples, leading to model degradation after training. To address this issue, the existing YOLO model uses a weight scaling coefficient to suppress the impact of negative samples on the loss function and increase the contribution of positive samples, thereby improving learning efficiency. However, simply assigning a value to the weight scaling coefficient can only be controlled within a certain threshold; otherwise, overfitting or training errors can occur.

[0245] Based on the initial loss function formula, it is found that loss conf Is the core that needs to be optimized, loss conf The following formula can be used:

[0246]

[0247] Taking the derivative of the above formula, we can get:

[0248]

[0249] Through the above formula, it is found that the derivative of confidence is positively correlated with the difference between the predicted value and the true value, so:

[0250]

[0251] Among them, ΔW represents the increment of the loss function to the variable, η W Represents the learning rate.

[0252] Based on the above three equations, we found that if the difference in the number of positive and negative samples is too large, the gradient features will be unclear, ultimately leading to detection failure. Based on this, we constructed the adaptive adjustment coefficient shown below to balance the impact of this difference.

[0253]

[0254] The adaptive adjustment coefficient can be generated adaptively according to the difference between the predicted value and the true value, where γ represents the base adjustment factor and b represents the exponential adjustment factor. b effectively controls the size of the adjustment coefficient by controlling the size of the exponential part, and effectively controls the size of α by using the high gradient change principle of the exponential function. When the sample under test is a positive sample is 1 when the sample is a negative sample is 0. After introducing the new adaptive scale factor, we have:

[0255]

[0256] in, Represents the confidence loss function term.

[0257] Taking the derivative of the above formula, we have:

[0258]

[0259] Therefore, after adding the adaptive scaling coefficient, the greater the difference between the predicted value and the true value, the greater the derivative of the loss term will be, and vice versa.

[0260] It can be found that when the value of b is too large, the gradient of positive and negative samples will change too much, while when the value of b is too small, the gradient of positive and negative samples will be insensitive to the change. When γ is 0, the loss function is consistent with the traditional loss function. Finally, after testing, the parameters are selected when b = 2 and γ = 2, which can be used to derive the formula for the loss function mentioned above.

[0261] In specific implementation, during the model training process, the detection accuracy of the detected pedestrian targets is expressed by the detection box accuracy and trajectory drawing accuracy.

[0262] The relevant indicators of detection accuracy are as follows.

[0263] Precision pp: This indicator indicates the accuracy of detecting individual pedestrian targets.

[0264] Recall rate r: This metric indicates how accurately the number of pedestrians is identified.

[0265] Accuracy a: This indicator indicates the accuracy of distinguishing pedestrians and other ground objects.

[0266] Among them, TP represents the number of positive samples predicted as positive samples, FP is the number of negative samples predicted as positive samples, FN is the number of positive samples that are mistakenly considered to be negative samples, and TN is the number of negative samples that are correctly considered to be negative samples.

[0267] The precision, recall and accuracy can be statistically plotted.

[0268] When the precision is greater than the precision threshold (e.g., 75%), the recall is greater than the recall threshold (e.g., 80%), and the accuracy is greater than the accuracy threshold (e.g., 85%), the model is trained and can be used.

[0269] You can set the precision threshold, recall threshold, and accuracy threshold according to actual needs.

[0270] During specific implementation, the image acquisition module 1 includes: an original frame image acquisition unit 11 and a splicing unit 12 .

[0271] The original frame image acquisition unit 11 is used to acquire multiple original frame images captured by multiple cameras at the same time point, wherein the shooting ranges of two adjacent cameras overlap or connect.

[0272] The stitching unit 12 is used to stitch a plurality of original frame images at the same time point to obtain a panoramic image, and use the panoramic image as the frame image.

[0273] In this embodiment, the original frame images captured by multiple cameras can be spliced ​​into a panoramic image, so that the activity trajectory of pedestrians can be analyzed in a larger range.

[0274] Alternatively, instead of splicing the original frame images, the frame images from multiple cameras can be analyzed separately to obtain multiple pedestrian trajectory analysis results. These multiple pedestrian trajectory analysis results are then combined for further processing. For example, after a pedestrian is detected in a certain frame, the pedestrian is numbered. Between frames, the same pedestrian is assigned an ID number based on their characteristics. The same person with the same characteristics has only one unique number. Pedestrians with the same ID number between different frames are considered the same person. The detection box here marks the pedestrian target individual, and the recorded information is the x-coordinate and y-coordinate relative to the upper left corner of the image, as well as the width w and height h of the box.

[0275] Example 3

[0276] This embodiment provides an image acquisition device. Figure 8-10 The diagram shows a front view, a left side view and a top view of an example of an image acquisition device, which includes a controller, a plurality of cameras (A, B, C and D), moving parts corresponding to the cameras and a bracket S.

[0277] The bracket S is provided with a slide rail (H1-5) corresponding to the camera, and the slide rail surrounds the central axis of the image acquisition device.

[0278] Each moving part includes: a rotating motor (J1-4) and a connecting rod (L1-4).

[0279] The controller is electrically connected to each rotating motor and each camera respectively.

[0280] The first end of the connecting rod is fixedly connected to the corresponding camera, and the second end of the connecting rod is transmission-connected to the corresponding rotating motor.

[0281] The rotary motor drives the first end of the corresponding connecting rod to rotate along the corresponding slide rail. The camera is located between two layers of slide rails. The rotary motor controls the camera's 360-degree rotation around the central axis along the slide rails via the connecting rod. Rotary motor J1 corresponds to connecting rod L1, camera A, and slide rails H1 and H2. Rotary motor J2 corresponds to connecting rod L2, camera B, and slide rails H2 and H3, and so on.

[0282] The controller is used to control the rotation motor for driving.

[0283] The controller is also used to receive frame images captured by the camera.

[0284] The frame image in step S11 in embodiment 1 may be derived from the frame image captured by the camera in this embodiment.

[0285] The controller can use the NVDIA Jetson TX2 model deep learning module, whose GPU (graphics processing unit) is equipped with 256 CUDA (graphics processing unit development environment) cores, and its CPU (central processing unit) uses the ARM architecture Quad ARM A57+HMP Dual Denver2 model, which can support 4k*2k 60Hz (hertz) video decoding and is equipped with 1 Gigabit Ethernet interface to meet data transmission and processing needs.

[0286] The number of cameras can be set according to actual needs.

[0287] Preferably, the number of cameras is 4, such as Figure 8-10 As shown in the figure, since the camera's field of view is 120 degrees and the entire circumference is 360 degrees, using four cameras can ensure at least a 25% overlap of viewing angles, while also reducing costs and avoiding wasted resources.

[0288] The bracket S can adopt a triangular support fixing method, or can adopt other support fixing methods such as four-corner support, base support, etc., which are not limited here.

[0289] In this embodiment, clock synchronization of multiple cameras may also be performed, as follows.

[0290] Figure 11 The diagram shows the clock synchronization principle. After the image acquisition device is turned on, the cameras are started (for example, the four-way signal switch jumps to position 2), and cameras A, B, C, and D begin working. The clocks inside the four cameras record the start-up time, where the start-up time of camera A is recorded as x1(t), the start-up time of camera B is recorded as x2(t), the start-up time of camera C is recorded as x3(t), and the start-up time of camera D is recorded as x4(t). Time synchronization will be carried out according to the following process:

[0291] (1) Time x1(t) passes through signal channel H1, and delay t1 is calculated as the correction value. At the same time, time x2(t) passes through signal channel H2 and is calculated with the calculated delay t1, and delay t2 is calculated using the time correction algorithm.

[0292] (2) The time correction algorithm uses the least mean square algorithm, using the signal input from the previous level as the reference signal and the signal input from the same level as the response signal. The square error is used instead of the mean square error. For example, the same level signal here is the signal passing through the signal channel H2 at time x2(t), and the signal input from the previous level is the delay t1 as the correction signal. The calculation formula of this algorithm is as follows:

[0293]

[0294] In the above formula, represents the gradient of a single squared error sequence, It represents the gradient of the statistical average of multiple square error sequences. The time correction algorithm adopted in the present invention uses the former to make an approximate estimate of the latter. Its recursive relationship expression is:

[0295]

[0296] In the above formula, w(n) represents the weight vector, μ represents the weight coefficient, and e(n) represents the error signal.

[0297] (3) Using the same process, after calculating the delay t2 as the correction value, the signal at time x3(t) through signal channel H3 is operated to calculate the delay t3.

[0298] (4) Based on the calculated delay t3 as the correction amount, the signal of time x4(t) through the signal channel H4 is operated to calculate the delay t4.

[0299] (5) Correct the calculated time delays t1, t2, t4, and t5 to the original time of each channel to obtain the final unified time signal.

[0300] (6) After obtaining a unified time signal, complete the clock synchronization operation (for example, turn the switch of the four-way signal to position 1), and formally start data collection for each camera.

[0301] The unified time signal enables multiple cameras to sample synchronously, and the frame images collected by each camera can form a correct correlation relationship on the timeline.

[0302] This embodiment can collect frame images through multiple cameras in the outdoor road environment of the community to realize the collection of pedestrian data in scenes with different perspectives. It can automatically detect pedestrians from video image data through deep learning methods, and draw detection boxes for pedestrians to mark them and the pedestrians' activity trajectories. It can investigate the patterns of population mobility in the community, and provide effective equipment and method support for community pedestrian data collection and pattern analysis in multi-perspective scenarios. At the same time, it provides a simpler design principle for multi-camera time synchronization, and has high detection accuracy for pedestrians and their activity trajectories, which is easy to implement.

[0303] In a specific implementation, the controller is further configured to obtain pedestrian trajectory analysis results for the current frame image within a preset time period. The pedestrian trajectory analysis results are obtained using the pedestrian trajectory analysis method described in Example 1. This analysis method can be executed by the controller in the image acquisition device or by other devices or apparatuses (e.g., a server), and the execution subject is not limited herein.

[0304] The controller is further configured to obtain pedestrian distribution data based on the pedestrian trajectory analysis results, determine a target scheduling pattern based on the distribution data and the matching relationship between the distribution data and the scheduling pattern, and control the rotation motor to drive the camera according to the target scheduling pattern so that the camera distribution conforms to the target scheduling pattern. The scheduling pattern includes the rotation direction and rotation angle of the camera.

[0305] The preset time period may be a preset time period starting from the start time of the image acquisition device.

[0306] The distribution data of pedestrians reflects the street layout, and the distribution data must conform to the street layout.

[0307] The angle between two adjacent cameras and the central axis and the position of the camera after the camera stops moving can be determined by the rotation direction and rotation angle of the camera.

[0308] The matching relationship between distribution data and dispatching methods (including the definition of dispatching methods) can be pre-set according to actual needs, so that the cameras can comprehensively monitor the streets and achieve cross-coverage of key areas.

[0309] Below Figure 8-10 Taking the image acquisition device as an example, an example of the working process of the image acquisition device is provided, including an example of camera scheduling.

[0310] Assumption: The camera is in a standard layout when powered on, such as Figure 12 As stated.

[0311] After the image acquisition device is turned on, it starts working when the time synchronization is completed. The four cameras A, B, C, and D are automatically driven by the rotating motor J1-4 and move to the specified position along the slide rail H1-5. That is, from the top view, they are located at the focus of the rectangular coordinate system and the slide rail. At the same time, the cameras start recording data synchronously.

[0312] The current application scenario is determined based on the frame image transmission rate. If the transmission rate is less than the rate threshold, the current application scenario is a small data scenario; if the transmission rate is greater than or equal to the rate threshold, the current application scenario is a large data scenario. It is understood that the transmission rate is related to the camera's image acquisition performance (for example, acquisition frequency, frame image resolution, etc.).

[0313] The rate threshold may be set according to actual needs, for example, to 800 Kbps (kilobits per second).

[0314] When the current application scenario is a big data scenario, it is assumed that the layout presets of the camera after the camera stops moving are divided into four types: I, II, III, and IV. Figure 13-16 The specific adjustment process is as follows:

[0315] (1) When pedestrian activities in the community are concentrated in one area, the camera layout adopts type I position, camera A maintains its initial position, the rotating motor J2 drives camera B to move between the slide rail H2 and the slide rail H3 through the connecting rod L2, and the rotating motor J3 drives camera C to move between the slide rail H3 and the slide rail H4 through the connecting rod L3, and the rotating motor J4 drives camera D to move between the slide rail H4 and the slide rail H5 through the connecting rod L4, and finally keeps the angle between camera A and camera D and the central axis within 90 degrees, and at the same time, cameras B and camera C are evenly distributed between them;

[0316] (2) When pedestrians in the community move regularly along a straight line, the camera placement adopts type II position. Camera A maintains its initial position. The rotating motor J2 drives the camera B to move between the slide rail H2 and the slide rail H3 through the connecting rod L2. At the same time, the rotating motor J3 drives the camera C to move between the slide rail H3 and the slide rail H4 through the connecting rod L3. The rotating motor J4 drives the camera D to move between the slide rail H4 and the slide rail H5 through the connecting rod L4. Finally, the angles between camera A and camera D and the central axis are kept within the range of 180 degrees. At the same time, cameras B and C are evenly distributed between them.

[0317] (3) When community pedestrians move on the opposite side of the outdoor visual unit layout position, the camera layout position adopts type III position, camera A maintains the initial position, the rotating motor J2 drives camera B to move between the slide rail H2 and the slide rail H3 through the connecting rod L2, and at the same time, the rotating motor J3 drives camera C to move between the slide rail H3 and the slide rail H4 through the connecting rod L3, and the rotating motor J4 drives camera D to move between the slide rail H4 and the slide rail H5 through the connecting rod L4, and finally keeps the angle between camera A and camera B and the central axis within 60 degrees, keeps the angle between camera C and camera D and the central axis within 60 degrees, and the distribution of cameras A, camera B and cameras C, camera D is symmetrical;

[0318] (4) When community pedestrians move around the outdoor visual unit, the camera layout adopts type IV position, camera A maintains the initial position, the rotating motor J2 drives camera B to move between the slide rail H2 and the slide rail H3 through the connecting rod L2, and the rotating motor J3 drives camera C to move between the slide rail H3 and the slide rail H4 through the connecting rod L3, and the rotating motor J4 drives camera D to move between the slide rail H4 and the slide rail H5 through the connecting rod L4, and finally the cameras are kept at 90 degrees between each other.

[0319] When the current application scenario is a small data scenario, from a bird's-eye view, with the center of the central axis as the origin O, the initial position of the four cameras OA represents the positive direction of the X axis, and OB represents the positive direction of the Y axis. After establishing the plane rectangular coordinate system, the specific operation process is as follows:

[0320] (1) When a pedestrian is first detected, the camera closest to the pedestrian is quickly moved to the direction of the line closest to the pedestrian, and the angles between the other three cameras and the camera closest to the pedestrian are kept at 60 degrees. Here, we take camera A as an example (e.g. Figure 17 As shown), if the pedestrian target appears in other directions, the process is similar;

[0321] (2) Camera A rotates and moves under the action of the connecting rod L1 driven by the rotating motor J1 until the viewing angle faces the target pedestrian. The direction of the plane coordinate system OX is taken as the starting direction, and the angle of rotation is calculated as α, that is, the size of the angle AOX is recorded as α. At this time, cameras B, C, and D all rotate and move in the direction of A;

[0322] (3) When α is less than 30 degrees, camera B rotates counterclockwise by (90-α-60) degrees under the drive of the rotating motor J2, camera C rotates counterclockwise by (60-α) degrees under the drive of the rotating motor J3, and camera D rotates clockwise by (60-α) degrees under the drive of the rotating motor J4;

[0323] (4) When α is greater than or equal to 30 degrees and less than 45 degrees, camera B rotates counterclockwise by (-90+α+60) degrees under the drive of the rotating motor J2, camera C rotates clockwise by (60-α) degrees under the drive of the rotating motor J3, and camera D rotates clockwise by (60-α) degrees under the drive of the rotating motor J4;

[0324] (5) When α is greater than 45 degrees and less than or equal to 90 degrees, it is considered to be in the first quadrant, and the camera B is closest to the pedestrian target. At this time, it is equivalent to quickly rotating camera B to adjust the viewing angle to be in a straight line with the pedestrian target, and then repeating the above steps (1) to (4).

[0325] During specific implementation, the image acquisition device may further include: a display screen, a communication module and a memory.

[0326] The controller is also electrically connected to the display screen, the communication module and the memory respectively.

[0327] The display can be used to display data sent by the controller, such as pedestrian trajectory analysis results. The controller can communicate with external devices (such as servers) through the communication module. The memory is used to store data sent by the controller and program data.

[0328] Example 4

[0329] Figure 18 This is a schematic diagram of the structure of an electronic device provided in Example 4 of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the pedestrian trajectory analysis method in Example 1 is implemented. Figure 18 The electronic device 30 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0330] The electronic device 30 may be a general-purpose computing device, such as a server device. Components of the electronic device 30 may include, but are not limited to, the at least one processor 31, the at least one memory 32, and a bus 33 connecting various system components (including the memory 32 and the processor 31).

[0331] The bus 33 includes a data bus, an address bus, and a control bus.

[0332] The memory 32 may include a volatile memory, such as a random access memory (RAM) 321 and / or a cache memory 322 , and may further include a read-only memory (ROM) 323 .

[0333] The memory 32 may also include a program / utility 325 having a set (at least one) of program modules 324, such program modules 324 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0334] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32 , such as the pedestrian trajectory analysis method in Example 1 of the present invention.

[0335] The electronic device 30 can also communicate with one or more external devices 34 (e.g., buttons, pointing devices, etc.). This communication can occur via an input / output (I / O) interface 35. Furthermore, the model-generated electronic device 30 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 36. As shown, the network adapter 36 communicates with other modules of the model-generated electronic device 30 via a bus 33. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the model-generated electronic device 30, including but not limited to microcode, device drivers, redundant processors, external disk drive arrays, RAID (RAID) systems, tape drives, and data backup storage systems.

[0336] It should be noted that although several modules / modules or submodules / modules of the electronic device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules / modules described above may be embodied in one module / module; conversely, the features and functions of one module / module described above may be further divided and embodied by multiple modules / modules.

[0337] Example 5

[0338] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the pedestrian trajectory analysis method in Embodiment 1 is implemented.

[0339] The readable storage medium may include, but is not limited to, a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0340] In a possible implementation manner, the present invention may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the pedestrian trajectory analysis method in Example 1.

[0341] The program code for executing the present invention may be written in any combination of one or more programming languages, and may be executed entirely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or entirely on the remote device.

[0342] Although specific embodiments of the present invention have been described above, those skilled in the art will appreciate that these are merely illustrative and that the scope of the present invention is defined by the appended claims. Those skilled in the art may make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, and such changes and modifications are intended to fall within the scope of the present invention.

Claims

1. A pedestrian trajectory analysis method, characterized in that: include: Get frame image; Inputting the frame image into a pedestrian trajectory analysis model to obtain a pedestrian trajectory analysis result; The pedestrian trajectory analysis result includes a marked image, which is an image in which each pedestrian in the frame image is marked with a detection box. The pedestrian trajectory analysis model is trained by inputting sample images including pedestrians marked with detection boxes into an improved YOLO model; The improved YOLO model replaces the feature extraction network of the existing YOLO model with selected layers of the deep learning neural network model; When the neural network model includes a ResNet-50 model, the improved YOLO model is divided into two types: a ResS-YOLO model and a ResB-YOLO model; In the ResB-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model; In the ResS-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model, and the 32nd to 40th layers of the ResNet-50 model are deleted; The pedestrian trajectory analysis model includes a first analysis model and a second analysis model; The first analysis model is obtained by training the ResS-YOLO model, and the second analysis model is obtained by training the ResB-YOLO model; The pedestrian trajectory analysis result also includes a trajectory line of each pedestrian, where the trajectory line is a line connecting the center points of the detection boxes corresponding to the same pedestrian in chronological order; The step of inputting the frame image into the pedestrian trajectory analysis model to obtain the pedestrian trajectory analysis result comprises: Acquiring a transmission rate of the frame image; When the transmission rate is less than a rate threshold, inputting the frame image into the first analysis model to obtain the marking image; When the transmission rate is greater than or equal to the rate threshold, the frame image is input into the second analysis model to obtain the marking image and the trajectory line.

2. The pedestrian trajectory analysis method according to claim 1, characterized in that: During the training process of the pedestrian trajectory analysis model, the loss function Loss is expressed by the following formula: ； S 2 =S w *S h ; ； Among them, N T Indicates the number of positive samples, N F represents the number of negative samples, represents the weight coefficient, i represents the grid number, j represents the corresponding detection unit number in the i-th grid, x and y represent the predicted horizontal and vertical coordinates of the center point of the detection box, respectively. and They represent the actual horizontal and vertical coordinates of the center point, w represents the predicted width of the detection box, and h represents the predicted height of the detection box. and Respectively represent the actual width and height of the detection box, B represents the number of the detection boxes, S 2 Indicates the total number of grids, S w Indicates the number of grids per row, S h Represents the number of grids in each column, C represents the confidence of the prediction of the detection box, Represents the actual confidence of the detection box, P represents the predicted object category probability, represents the actual object category probability, represents the probability of the target object existing in the grid cell, Indicates the overlap rate between the detection box predicted by the model and the actual object; When the detection unit is a positive sample, is 1, is 0; when the detection unit is a negative sample, is 0, is 1.

3. The pedestrian trajectory analysis method according to claim 1, characterized in that: The step of acquiring the frame image comprises: Acquire multiple original frame images captured by multiple cameras at the same time point; wherein the shooting ranges of two adjacent cameras overlap or connect; The multiple original frame images at the same time point are stitched together to obtain a panoramic image, and the panoramic image is used as the frame image.

4. A pedestrian trajectory analysis system, characterized in that: include: Image acquisition module and analysis result generation module; The image acquisition module is used to acquire frame images; The analysis result generating module is used to input the frame image into the pedestrian trajectory analysis model to obtain the pedestrian trajectory analysis result; The pedestrian trajectory analysis result includes a marked image, which is an image in which each pedestrian in the frame image is marked with a detection box. The pedestrian trajectory analysis model is trained by inputting sample images including pedestrians marked with detection boxes into an improved YOLO model; The improved YOLO model replaces the feature extraction network of the existing YOLO model with selected layers of the deep learning neural network model; When the neural network model includes a ResNet-50 model, the improved YOLO model is divided into two types: a ResS-YOLO model and a ResB-YOLO model; In the ResB-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model; In the ResS-YOLO model, the 49th layer of the ResNet-50 model is connected to the detection network of the existing YOLO model, and the 32nd to 40th layers of the ResNet-50 model are deleted; The pedestrian trajectory analysis model includes a first analysis model and a second analysis model; The first analysis model is obtained by training the ResS-YOLO model, and the second analysis model is obtained by training the ResB-YOLO model; The pedestrian trajectory analysis result also includes a trajectory line of each pedestrian, where the trajectory line is a line connecting the center points of the detection boxes corresponding to the same pedestrian in chronological order; The analysis result generating module includes: a transmission rate acquiring unit, a first analysis generating unit and a second analysis generating unit; The transmission rate acquisition unit is used to acquire the transmission rate of the frame image; The first analysis and generation unit is configured to input the frame image into the first analysis model to obtain the marking image when the transmission rate is less than a rate threshold; The second analysis generation unit is configured to input the frame image into the second analysis model to obtain the marking image and the trajectory line when the transmission rate is greater than or equal to the rate threshold.

5. An image acquisition device, characterized in that: include: A controller, a plurality of cameras, a moving part corresponding to the cameras, and a bracket; The bracket is provided with a slide rail corresponding to the camera, and the slide rail surrounds the central axis of the image acquisition device; Each of the moving parts includes: a rotating motor and a connecting rod; The controller is electrically connected to each of the rotating motors and each of the cameras respectively; The first end of the connecting rod is fixedly connected to the corresponding camera, and the second end of the connecting rod is transmission-connected to the corresponding rotating motor; The rotary motor is used to drive the first end of the corresponding connecting rod to rotate along the corresponding slide rail; The controller is used to control the rotating motor to drive; The controller is also used to receive the frame image collected by the camera; The controller is further configured to obtain a pedestrian trajectory analysis result of a current frame image within a preset time period; wherein the pedestrian trajectory analysis result is obtained using the pedestrian trajectory analysis method according to any one of claims 1 to 3; The controller is also used to obtain pedestrian distribution data based on the pedestrian trajectory analysis results, and obtain a target scheduling method based on the distribution data and the matching relationship between the distribution data and the scheduling method, and control the rotating motor to drive according to the target scheduling method so that the distribution of the camera conforms to the target scheduling method; wherein, the scheduling method includes the rotation direction and rotation angle of the camera.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the pedestrian trajectory analysis method according to any one of claims 1 to 3 is implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the pedestrian trajectory analysis method according to any one of claims 1 to 3 is implemented.

Images