Speed ​​detection method and electronic device

Abstract

The present application discloses a speed detection method and electronic device, which obtain a second surveillance video containing a floor tile area, and determine the image position information of multiple corner points of the floor tile area in multiple second images in the second surveillance video. According to the image position information of the multiple corner points and the position information of the corresponding world coordinate system, the image acquisition device is calibrated to determine the correction parameters of the image acquisition device. When performing speed detection, after determining the initial detection frames of different persons in the multiple first images in the first surveillance video, the initial detection frames are corrected by the correction parameters to obtain the corresponding target detection frames. Finally, the speed information of different persons is determined based on the timestamp information of the multiple first images and the position information of the target detection frames of the different persons. The technical solution protected by the present application has the characteristics of high accuracy and fast inference speed, and meets the characteristics of trustworthiness.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a speed detection method and electronic equipment. Background Art

[0002] With my country's rapid economic growth and rapid urbanization, the country's urban rail transit industry has also experienced rapid development. With technological advancements, intelligent rail transit is the current development direction of the industry. Intelligent rail transit involves deploying cameras in subway stations to analyze scene events, providing relevant information to personnel and preventing dangerous incidents.

[0003] Passenger flow detection is to detect and analyze the passenger flow speed in places where passenger flow gathers, such as security checks and gates, and calculate the average passenger speed in a designated area to determine whether the current channel is blocked, so that staff can find out the situation in time and ensure smooth passenger flow.

[0004] Existing methods typically use depth cameras or laser ranging sensors to determine the correspondence between actual distance and image pixels, or construct a mapping matrix based on the camera's internal and external parameters to detect passenger flow speed. Solutions based on depth cameras or laser ranging sensors have high hardware costs, while those based on the camera's internal and external parameters require customizing large calibration plates, making implementation complex and labor-intensive. Summary of the Invention

[0005] The embodiments of the present application provide a speed detection method and electronic device to solve the problems of high cost and difficulty in implementation of passenger flow speed detection solutions in the prior art.

[0006] In a first aspect, the present application provides a speed detection method, the method comprising:

[0007] In response to a passenger flow speed detection instruction, determining initial detection frames of different persons in a plurality of first images in a first surveillance video captured by an image acquisition device, and correcting the initial detection frames using a predetermined correction parameter of the image acquisition device to obtain corresponding target detection frames;

[0008] determining speed information of the different persons according to the timestamp information of the multiple first images and the position information of the target detection frames of the different persons;

[0009] Among them, the process of determining the correction parameters of the image acquisition device is to obtain multiple second images in the second surveillance video containing the floor tile area; for the multiple second images, determine the image position information of multiple corner points of the floor tile area in the second image; based on the image position information of the multiple corner points in the multiple second images, and the pre-saved position information of the world coordinate system corresponding to the multiple corner points, determine the correction parameters of the image acquisition device.

[0010] In a second aspect, the present application provides a speed detection device, comprising:

[0011] a correction module, configured to determine, in response to a passenger flow speed detection instruction, initial detection frames of different persons in a plurality of first images in a first surveillance video captured by an image acquisition device, and correct the initial detection frames using a predetermined correction parameter of the image acquisition device to obtain corresponding target detection frames;

[0012] a first determining module, configured to determine speed information of the different persons based on timestamp information of the plurality of first images and position information of target detection frames of the different persons;

[0013] The second determination module is used to obtain multiple second images in a second surveillance video containing a floor tile area; determine the image position information of multiple corner points of the floor tile area in the second images for the multiple second images; and determine the correction parameters of the image acquisition device based on the image position information of the multiple corner points in the multiple second images and the pre-saved position information of the world coordinate system corresponding to the multiple corner points.

[0014] In a third aspect, the present application provides an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0015] Memory for storing computer programs;

[0016] The processor is used to implement the method steps when executing the program stored in the memory.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described are implemented.

[0018] The above technical solution has the following advantages or beneficial effects:

[0019] The present application pre-acquires a second surveillance video containing a floor tile area, and determines the image position information of multiple corner points of the floor tile area in multiple second images in the second surveillance video. Because the position of the floor tile area is fixed, the position information of the world coordinate system corresponding to the multiple corner points of the floor tile area can be pre-saved. According to the image position information of the multiple corner points and the position information of the corresponding world coordinate system, the image acquisition device is calibrated to determine the correction parameters of the image acquisition device. When performing passenger flow speed detection, after determining the initial detection frames of different people in the multiple first images in the first surveillance video, the initial detection frames are corrected by the correction parameters to obtain the corresponding target detection frames. Finally, the speed information of different people is determined based on the timestamp information of the multiple first images and the position information of the target detection frames of the different people. The present application does not require the customization of a large-area calibration plate, nor does it require the configuration of a depth camera or a laser ranging sensor, thus solving the problem that the passenger flow speed detection solution of the prior art is high in cost and difficult to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 A schematic diagram of the first speed detection process provided by this application;

[0022] Figure 2 A schematic diagram of the second speed detection process provided by this application;

[0023] Figure 3 A schematic diagram of the third speed detection process provided by this application;

[0024] Figure 4 A schematic diagram of the fourth speed detection process provided by this application;

[0025] Figure 5 Schematic diagrams of on-site floor tiles in subways, squares, etc. provided for this application;

[0026] Figure 6 Schematic diagram of the corner points of the floor tile area provided for this application;

[0027] Figure 7 The overall flow chart of passenger flow speed detection provided for this application;

[0028] Figure 8 Schematic diagram of the camera calibration process provided for this application;

[0029] Figure 9 Schematic diagram of the walkable area segmentation provided for this application;

[0030] Figure 10 Schematic diagram of the crude extraction results provided for this application;

[0031] Figure 11 A schematic diagram of a local area of ​​a corner point provided for this application;

[0032] Figure 12 Schematic diagram of the fine extraction process provided for this application;

[0033] Figure 13 Schematic diagram of Hough line detection provided for this application;

[0034] Figure 14 Schematic diagram of the straight line detection effect provided by this application;

[0035] Figure 15 Schematic diagram of the feature point extraction effect provided by this application;

[0036] Figure 16 Schematic diagram of the corner point coordinate distribution provided for this application;

[0037] Figure 17 Schematic diagram of coordinate system conversion during camera calibration provided in this application;

[0038] Figure 18 The coordinate transformation relationship provided for this application;

[0039] Figure 19 A schematic diagram of pedestrians near the detection line provided for this application;

[0040] Figure 20 Schematic diagram of pedestrian head detection provided for this application;

[0041] Figure 21 Schematic diagram of the process of determining pedestrian movement speed provided by this application;

[0042] Figure 22 This is a schematic diagram of the positions of the detection frames for the same ID provided in this application;

[0043] Figure 23 Schematic diagram of the relationship between the speed, the vertical coordinate of the target in the image, and the movement direction of the pedestrian provided by this application;

[0044] Figure 24 Schematic diagram of the relationship between the speed and the vertical coordinate of the target in the image provided by this application;

[0045] Figure 25 A schematic diagram showing the relationship between speed and pedestrian movement direction provided for this application;

[0046] Figure 26 Schematic diagram of subway scene test provided for this application;

[0047] Figure 27 A schematic diagram of the structure of the passenger flow speed detection device provided in this application;

[0048] Figure 28 This is a schematic diagram of the electronic device structure provided in this application. DETAILED DESCRIPTION

[0049] In order to make the purpose and implementation of this application clearer, the exemplary implementation of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of the embodiments.

[0050] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0051] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0052] The terms "comprise," "include," and "have," and any variations thereof, are intended to cover but not exclude inclusion; for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0053] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functionality associated with that element.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

[0055] For ease of explanation, the above description has been made with reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations are possible. The above embodiments are selected and described to better explain the principles and practical applications, so that those skilled in the art can better utilize the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0056] Figure 1 This is a schematic diagram of the speed detection process provided by this application. The process includes the following steps:

[0057] S101: In response to a passenger flow speed detection instruction, determine the initial detection frames of different persons in multiple first images in a first surveillance video captured by an image acquisition device, and correct the initial detection frames using a predetermined correction parameter of the image acquisition device to obtain a corresponding target detection frame; the correction parameter determination process of the image acquisition device is to obtain multiple second images in a second surveillance video containing a floor tile area; for the multiple second images, determine the image position information of multiple corner points of the floor tile area in the second images; determine the correction parameters of the image acquisition device based on the image position information of the multiple corner points in the multiple second images and the pre-saved position information of the world coordinate system corresponding to the multiple corner points.

[0058] S102: Determine speed information of the different persons according to the timestamp information of the multiple first images and the position information of the target detection frames of the different persons.

[0059] The speed detection method provided in this application is applied to an electronic device, which may be a PC, a tablet computer, or a server.

[0060] This application takes into account that most sites such as subways and squares use standardized floor tiles. The floor tiles are fixed in size, evenly distributed in the image, and occupy almost the vast majority of the image. They can serve as highly simulated large-scale "calibration plates" to support on-site calibration of image acquisition devices. The image acquisition device can be a camera, a video camera, etc. Thus, the correction parameters (internal and external parameters) of the image acquisition device are obtained, and a more accurate mapping matrix is ​​constructed. Through this mapping matrix, the accuracy of pedestrian speed estimation is effectively improved.

[0061] For example, consider an image acquisition device such as a surveillance camera. A surveillance camera is deployed in a surveillance scene, and the electronic device can capture a second surveillance video containing a floor tile area through the camera. Each second image in the second surveillance video can be obtained through frame processing. For each second image, a region segmentation algorithm can be used to determine the floor tile area, wall area, column area, and so on within the second image. After determining the floor tile area within the second image, a target detection algorithm can be used to determine multiple corner points within the floor tile area. Corner points refer to the corner points where adjacent floor tiles meet. After determining the multiple corner points within the floor tile area, image position information for the multiple corner points can be obtained.

[0062] The floor tiles are of fixed size, and the electronic device can pre-store the world coordinate system position information corresponding to each corner point. It should be noted that the origin of the world coordinate system corresponds to the origin of the image coordinate system. For example, if the corner point in the lower left corner of the image is the origin, then the coordinates of the lower left corner point in the world coordinate system are also (0, 0). After determining the image position information of the multiple corner points in the multiple second images, the calibration parameters of the camera that captured the second surveillance video can be determined based on the image position information of the multiple corner points in the multiple second images and the world coordinate system position information corresponding to the multiple corner points.

[0063] Optionally, the camera calibration parameters are determined according to the Zhang Zhengyou calibration method. The camera calibration algorithm inputs the image position information of multiple corner points in multiple second images and the position information of the world coordinate system corresponding to the multiple corner points, and solves and optimizes the internal parameters, external parameters and distortion parameters. Camera calibration is mainly based on the camera imaging model, and its essence is the transformation relationship between the world coordinate system to the camera coordinate system to the image coordinate system to the pixel coordinate system. Relying on this transformation relationship, a model between the pixel coordinate system and the world coordinate system is established, and the camera parameters are finally optimized. In the same scene, there is no need to calibrate the image acquisition device every time a speed detection is performed. It is only necessary to save the correction parameters of the image acquisition device after the calibration is completed. Each time a passenger flow detection is performed, the saved correction parameters are obtained and used for calculation.

[0064] Upon receiving a passenger flow speed detection instruction, the electronic device identifies each first image in the first surveillance video captured by the image acquisition device and, using a target detection algorithm, determines an initial detection frame for each person in each first image. The initial detection frame for each person is corrected using the correction parameters of the image acquisition device to obtain a target detection frame corresponding to the initial detection frame. Finally, for each first image, the travel distance and time taken by each person are determined based on the timestamp information of the first image and the position information of the target detection frame for each person in the first image. The speed information of each person is then determined based on the ratio of the travel distance to the time taken.

[0065] The present application pre-acquires a second surveillance video containing a floor tile area, and determines the image position information of multiple corner points of the floor tile area in multiple second images in the second surveillance video. Because the position of the floor tile area is fixed, the position information of the world coordinate system corresponding to the multiple corner points of the floor tile area can be pre-saved. According to the image position information of the multiple corner points and the position information of the corresponding world coordinate system, the image acquisition device is calibrated to determine the correction parameters of the image acquisition device. When performing passenger flow speed detection, after determining the initial detection frames of different people in the multiple first images in the first surveillance video, the initial detection frames are corrected by the correction parameters to obtain the corresponding target detection frames. Finally, the speed information of different people is determined based on the timestamp information of the multiple first images and the position information of the target detection frames of the different people. The present application does not require the customization of a large-area calibration plate, nor does it require the configuration of a depth camera or a laser ranging sensor, thus solving the problem that the passenger flow speed detection solution of the prior art is high in cost and difficult to implement.

[0066] Reliable feature 1: High accuracy in measuring pedestrian speed. Existing methods all use depth cameras or laser ranging sensors to obtain the correspondence between the actual distance and the pixels in the image, and this solution has high hardware costs. Alternatively, based on the calibration of the camera, internal and external parameters are obtained to construct a mapping matrix to detect passenger flow speed. This method relies on camera calibration and measurement of installation angles and positions, making it difficult to implement. This solution regards the tile pattern as a large "calibration plate" and accurately obtains the corner point positions through a series of corner point extractions, thereby performing on-site calibration of the camera. At the same time, the speed value is corrected and compensated based on the combing model to reduce the impact of measurement errors, which can greatly improve the accuracy of speed measurement.

[0067] Trustworthy Feature 2: Easy engineering implementation and high real-time performance. Automated extraction of tile corner points through image segmentation and object detection significantly improves the algorithm's automation and real-time performance. Furthermore, real-time calculation of pedestrian distances from the detection line allows for screening of targets with desired speeds, significantly reducing computational complexity and improving the algorithm's real-time performance.

[0068] Considering the problem that pedestrians block corner points of the ground, resulting in inaccurate corner point extraction, in this application, for the multiple second images, determining the image position information of the multiple corner points of the floor tile area in the second images includes:

[0069] performing person detection on the plurality of second images, and selecting a second image in which no person is present as a third image;

[0070] The floor area sub-image in the plurality of third images is identified by a target detection algorithm, and image position information of a plurality of corner points of the tile area sub-image is determined.

[0071] After the electronic device acquires the second surveillance video and identifies multiple second images, it uses a target detection algorithm to detect people in the multiple second images, filters out the second images containing people, and selects the second images without people as the third images. The target detection algorithm then identifies the floor area sub-image in each frame of the third image, and finally determines the image position information of multiple corner points of the tile area sub-image using the target detection algorithm.

[0072] Optionally, after the electronic device obtains the second surveillance video and determines the second image of each frame, it can also first determine the floor area sub-image in each frame of the second image, and then detect whether there is a person in the floor area sub-image, and use the second image in which there is no person in the floor area sub-image as the third image, and then determine the image position information of multiple corner points of the brick area sub-image in each frame of the third image.

[0073] Figure 2 The speed detection process provided in this application includes the following steps:

[0074] S201: Acquire multiple second images in a second surveillance video containing a floor tile area; perform personnel detection on the multiple second images, and select the second image in which no personnel is present as the third image; identify the floor area sub-image in the multiple third images through a target detection algorithm, and determine the image position information of multiple corner points of the tile area sub-image.

[0075] S202: Determine correction parameters of the image acquisition device according to image position information of multiple corner points in the multiple second images and pre-stored position information of a world coordinate system corresponding to the multiple corner points.

[0076] S203: In response to the passenger flow speed detection instruction, determine the initial detection frames of different persons in multiple first images in the first surveillance video captured by the image acquisition device, and correct the initial detection frames using the predetermined correction parameters of the image acquisition device to obtain corresponding target detection frames.

[0077] S204: Determine speed information of the different persons according to the timestamp information of the multiple first images and the position information of the target detection frames of the different persons.

[0078] In order to make the determination of the image position information of the multiple corner points more accurate, in this application, determining the image position information of the multiple corner points of the tile area sub-image includes:

[0079] Inputting the tile area sub-image into a trained corner detection network model, and determining a plurality of corner regions in the tile area sub-image based on the corner detection network model;

[0080] For the multiple corner point regions, line detection and feature point extraction are performed on the corner point regions to determine image position information of the corner points in the corner point regions.

[0081] The electronic device stores a trained corner detection network model. The corner detection network model is trained based on sample images and corresponding label information in a training set. The label information is the image location information of corner regions in the sample images. The trained corner detection network model is input into a brick region sub-image. Based on the corner detection network model, multiple corner regions in the brick region sub-image are determined.

[0082] For multiple corner regions, line detection and feature point extraction are performed on the corner regions to determine the image position information of the corner points in the corner regions. Line detection can be performed on the corner regions using the Hough line detection algorithm, and feature points in the corner regions can be determined using a feature point extraction algorithm. Feature points can be corner points, edge intersections, spots, etc. Feature points can be accurately detected in different images and are robust to image changes such as rotation, scaling, and brightness changes. Feature point extraction algorithms include the Harris corner detection algorithm, SIFT algorithm, SURF algorithm, and FAST algorithm.

[0083] Figure 3 This is a schematic diagram of the speed detection process provided by this application. The process includes the following steps:

[0084] S301: Acquire multiple second images in a second surveillance video containing a floor tile area; perform personnel detection on the multiple second images, and select the second image in which no person is present as the third image; and identify the floor area sub-image in the multiple third images through a target detection algorithm.

[0085] S302: Input the brick area sub-image into a trained corner detection network model, and determine multiple corner point areas in the brick area sub-image based on the corner detection network model; perform straight line detection and feature point extraction on the multiple corner point areas to determine the image position information of the corner points in the corner point areas.

[0086] S303: Determine correction parameters of the image acquisition device according to image position information of multiple corner points in the multiple second images and pre-stored position information of a world coordinate system corresponding to the multiple corner points.

[0087] S304: In response to the passenger flow speed detection instruction, determine the initial detection frames of different persons in multiple first images in the first surveillance video captured by the image acquisition device, and correct the initial detection frames using the predetermined correction parameters of the image acquisition device to obtain corresponding target detection frames.

[0088] S305: Determine speed information of the different persons according to the timestamp information of the multiple first images and the position information of the target detection frames of the different persons.

[0089] In this application, performing line detection and feature point extraction on the corner point area to determine image position information of the corner points in the corner point area includes:

[0090] Performing line detection on the corner point area, and if there are two straight lines in the corner point area, determining the image position information of the corner point according to the intersection of the two straight lines;

[0091] If there is a straight line in the corner point area, extract feature points from the corner point area, and determine the image position information of the corner point based on the feature point closest to the straight line;

[0092] If there is no straight line in the corner point area, it is determined that there is no corner point in the corner point area.

[0093] Case 1: There are two straight lines in the corner area.

[0094] If two straight lines are detected in the corner point area, feature point extraction may be omitted and the image position information of the corner point may be determined directly based on the intersection of the two straight lines. Alternatively, the image position information of the intersection of the two straight lines may be directly used as the image position information of the corner point.

[0095] Preferably, determining the image position information of the corner point according to the intersection of the two straight lines includes:

[0096] If the intersection of the two straight lines is within the corner point area, the image position information of the intersection is used as the image position information of the corner point; if the intersection of the two straight lines is not within the corner point area, feature points are extracted from the corner point area, and the image position information of the corner point is determined based on the feature point closest to the two straight lines.

[0097] Based on the image position information of the intersection of the two straight lines and the image position information of the corner point area, it can be determined whether the intersection of the two straight lines is within the corner point area. If so, the image position information of the intersection is used as the image position information of the corner point, and there is no need to extract feature points. If not, feature point extraction is performed on the corner point area, and the distance between each feature point and the two straight lines is determined respectively. The feature point corresponding to the closest distance is selected as the corner point to obtain the image position information of the corner point.

[0098] Preferably, if the closest distance is less than a preset distance threshold, the image position information of the feature point corresponding to the closest distance is used as the image position information of the corner point; otherwise, it is determined that no corner point exists in the corner point area.

[0099] Case 2: There is a straight line in the corner area.

[0100] At this time, feature points are extracted from the corner point area, the distance between each feature point and the straight line is determined respectively, and the feature point corresponding to the closest distance is selected as the corner point to obtain the image position information of the corner point.

[0101] Specifically, determining the image position information of the corner point according to the feature point closest to the straight line includes:

[0102] If the shortest distance between the feature point and the straight line is less than a preset distance threshold, the image position information of the feature point corresponding to the shortest distance is used as the image position information of the corner point; otherwise, it is determined that no corner point exists in the corner point area.

[0103] Case 3: There is no straight line in the corner area.

[0104] There is no need to extract feature points, and it is directly determined that there are no corner points in the corner point area.

[0105] In order to further make the determined image position information of the corner points more accurate, after determining the image position information of multiple corner points of the floor tile area in the second image, the method further includes:

[0106] For the multiple corner points, determine multiple candidate corner points adjacent to the corner points;

[0107] Determining the distances between the corner point and the multiple candidate corner points respectively, and determining missed detection position information corresponding to the falsely detected corner points and the missed detection corner points according to the distances between the corner point and the multiple candidate corner points;

[0108] Corner points are supplemented in the missed detection position information, and the falsely detected corner points are deleted.

[0109] For multiple corner points, determine multiple candidate corner points adjacent to the corner point, where the number of candidate corner points is less than or equal to 4. Determine the distances between the corner point and the multiple candidate corner points. Based on the distances between the corner point and the multiple candidate corner points, determine the missed detection position information corresponding to the falsely detected corner points and the missed detection corner points. Optionally, if a distance is significantly longer than approximately twice the distances of the other distances, it is considered that a missed detection corner point exists in the direction corresponding to that distance. The missed detection position information of the missed detection corner point is the center position information of the line connecting the corner point and the candidate corner points in that direction. If a distance is significantly shorter than the distances in all other directions, the corner point in that direction is considered to be a falsely detected corner point and is directly deleted.

[0110] Considering that the more targets there are, the greater the computational complexity of the target tracking algorithm, and the lower the efficiency and real-time performance of event recognition. In this application, determining the initial detection frames of different persons in multiple first images in the first surveillance video captured by the image acquisition device includes:

[0111] Determine the initial head frames of different people in multiple first images in the first surveillance video captured by the image acquisition device; determine the distance between the initial head frame and a preset detection line, determine the initial head frame with a distance less than a preset distance threshold as the initial detection frame, and filter out the initial head frame with a distance not less than the preset distance threshold.

[0112] Due to the influence of the focal length of the image acquisition device, the distance of the target pedestrian in the first image appears larger when closer and smaller when farther away. Therefore, it is proposed to determine whether the pedestrian is near the detection line based on the pedestrian's head frame. If the pedestrian is far away from the detection line and is not near the detection line, target tracking will no longer be performed for this target. Otherwise, the target is considered to be in the area near the set detection line and target tracking is started. Multi-target tracking algorithms can be divided into three categories: algorithms that do not use feature information matching (such as SORT and Bytetrack algorithms), algorithms that use feature information matching (such as deepsort algorithms), and algorithms that output feature information simultaneously during detection (such as JDE and FairMOT algorithms). These algorithms are classified according to the matching method between different frames for the same target.

[0113] Considering that the moving distance of a person is affected by the focal length and distortion of the image, the pedestrians in areas with large distortion move faster, and pedestrians in nearby areas move faster, resulting in inaccurate speed calculation. In this application, after determining the speed information of the different people, the method further includes:

[0114] Fitting a first parameter based on a relationship between the speed information of the different persons and the vertical coordinate of the target detection frame;

[0115] According to the relationship between the speed information and the moving direction angles of the different persons, a second parameter is obtained by fitting;

[0116] The speed information of the different persons is updated according to the first parameter and the second parameter.

[0117] By counting the speeds of a large number of pedestrians, we found that speed is primarily related to the target's vertical coordinate in the image and the pedestrian's direction of movement. The speed information of different people and the vertical coordinate of the target detection frame are linearly related. Using the speed = Ay model, we obtain the first parameter A, where speed represents speed and y represents the vertical coordinate.

[0118] The speed information of different people is inversely related to the moving direction angle. The speed=Csin(angle) model is fitted to obtain the second parameter C, where speed is the speed, angle is the moving direction angle, and sin is the sine function.

[0119] The speed information of the different persons is updated according to the first parameter and the second parameter. The update is as follows:

[0120] speed=(Distance_sum / △T) / Ay-Csin(angle);

[0121] speed=speed<0?0:speed.

[0122] Distance_sum is the moving distance, and △T is the moving time.

[0123] Figure 4 The speed detection process provided in this application includes the following steps:

[0124] S401: Acquire multiple second images in a second surveillance video containing a floor tile area; perform personnel detection on the multiple second images, and select the second image in which no person is present as the third image; and identify the floor area sub-image in the multiple third images through a target detection algorithm.

[0125] S402: Input the brick area sub-image into a trained corner detection network model, and determine multiple corner point regions in the brick area sub-image based on the corner detection network model; perform straight line detection and feature point extraction on the multiple corner point regions to determine the image position information of the corner points in the corner point regions.

[0126] S403: Determine correction parameters of the image acquisition device according to image position information of multiple corner points in the multiple second images and pre-stored position information of a world coordinate system corresponding to the multiple corner points.

[0127] S404: In response to the passenger flow speed detection instruction, determine the initial detection frames of different persons in multiple first images in the first surveillance video captured by the image acquisition device, and correct the initial detection frames using the predetermined correction parameters of the image acquisition device to obtain corresponding target detection frames.

[0128] S405: Determine speed information of the different persons according to the timestamp information of the multiple first images and the position information of the target detection frames of the different persons.

[0129] S406: Fitting a first parameter based on the relationship between the speed information of the different people and the vertical coordinate of the target detection frame; fitting a second parameter based on the relationship between the speed information of the different people and the moving direction angle; updating the speed information of the different people based on the first parameter and the second parameter.

[0130] The passenger flow speed detection process of the present application is described in detail below with reference to the accompanying drawings.

[0131] This application proposes a simple and efficient method for detecting passenger flow speed. Standardized floor tiles are often used in places like subways and squares. These tiles are of fixed size, evenly distributed in the image, and occupy almost the entire image. They can serve as highly simulated large-scale "calibration plates" to support on-site camera calibration. This allows the camera's internal and external parameters to be acquired, allowing for a more accurate mapping matrix to be constructed. This mapping matrix effectively improves the accuracy of pedestrian speed estimation. Furthermore, this application proposes a method for secondary correction of speed values ​​based on mathematical statistics, significantly improving the accuracy of passenger flow speed detection. Figure 5 Schematic diagram of on-site floor tiles in subways, squares, etc. provided for this application. Figure 6 Schematic diagram of the corner points of the floor tile area provided for this application. Figure 6 The dots in the middle are corner points.

[0132] Figure 7 The overall flow chart of passenger flow speed detection provided for this application includes input image, image acquisition device calibration, target detection and tracking, calculation of passenger flow speed, event recognition and abnormal events.

[0133] The video stream of the image acquisition device is decoded to obtain each frame of image to be processed.

[0134] Image acquisition device (camera) calibration: This step is crucial for accurately determining the corner locations of floor tiles. First, a pedestrian detection algorithm is used to capture an image without pedestrians. An image segmentation algorithm is then used to extract the area containing only the floor. Feature point extraction and the Hough line detection algorithm are then used to extract the corners of the tiles. This constructs an image sequence of corner points and a world coordinate matrix. Finally, the camera calibration parameters are calculated through iteration.

[0135] Target detection and tracking: After camera calibration, pedestrian detection is performed on each frame using the target detection algorithm, and the detected pedestrian box and confidence score are output. For each pedestrian box, feature matching is performed and N frames are tracked continuously. The historical trajectory of each target ID and the corresponding tracking time are obtained based on the tracking algorithm.

[0136] Calculate passenger flow speed: Use camera calibration to obtain the camera's distortion parameters and internal and external parameters to calculate the speed value of each pedestrian. Based on a mathematical and statistical model, the speed value is corrected twice in combination with the pedestrian's position and movement direction to obtain the accurate passenger flow speed.

[0137] Event recognition: When monitoring pedestrians whose moving speed exceeds the set threshold, the system circles the target and outputs a video clip of the event, generating an alarm to alert business personnel.

[0138] Figure 8 This is a schematic diagram of the camera calibration process provided in this application. Figure 8As shown, pedestrian detection: For the input video stream, video decoding is performed to obtain an image sequence, image1, image2, image3, ...imagen... A target detection algorithm is used to detect pedestrian targets in each frame of the image sequence. Pedestrian target detection can be performed using common target detection methods such as the Faster RCNN series, the YOLO series, the EfficientDet series, and other portrait or head detection algorithms. The pedestrian target detection algorithm outputs the location and confidence of the pedestrian bounding box. If the algorithm outputs a pedestrian bounding box, it is assumed that a pedestrian is present in the current frame, and there is a high probability that the pedestrian will occlude the corner of the brick on the ground, so detection continues to the next frame. Conversely, if the algorithm does not output a pedestrian bounding box, it is assumed that there is no pedestrian in the current frame, and the current frame image is cached.

[0139] Image segmentation of walkable areas: The main goal of image segmentation is to obtain walkable areas and avoid interference from non-ground corners. At the same time, in order to minimize the influence of external environments such as light, multiple images without pedestrians are cached to prepare for subsequent corner detection. The total number of cached images is counted in real time. If the number exceeds the set threshold m, the image segmentation module is started to perform image segmentation of the walkable area on the cached images. The segmentation effect is as follows. Image segmentation algorithms can include but are not limited to threshold-based segmentation algorithms, edge, clustering, graph theory or deep learning segmentation algorithms, such as FCN, U-Net, etc.

[0140] Figure 9 Schematic diagram of the walkable area segmentation provided for this application.

[0141] Corner extraction: The corners of bricks in actual images often contain noise, blur, occlusion, and deformation, leading to inaccurate corner position and number. First, we perform a rough corner extraction based on an object detection algorithm to obtain a local image of the corners. Then, we refine this local image using feature point extraction and the Hough line detection algorithm. Further corner point screening ultimately leads to accurate corner coordinates.

[0142] Crude extraction:

[0143] Input the walkable area image into an existing object detection algorithm to obtain the box position and confidence score for each corner point. Low-confidence boxes are filtered out, ultimately obtaining a sequence of local images of the corner points {ROI_1, ROI_2, ROI_3…ROI_n}, as shown in the accompanying figure. Object detection algorithms can use common deep learning methods such as the Faster RCNN series, the YOLO series, and the EfficientDet series, and can be trained using a large number of annotated images. Figure 10 Schematic diagram of the crude extraction results provided for this application. Figure 11 Schematic diagram of the local area of ​​the corner point provided in this application.

[0144] Fine extraction:

[0145] After the coarse positioning of the corner points, since the positions of the corner points are at the pixel level, there will be a certain deviation from the actual corner point positions, so fine positioning is required. This step mainly uses Hough line detection and feature point extraction to refine the corner point extraction of the local image sequence of the corner points to obtain more accurate corner point position information.

[0146] Figure 12 This is a schematic diagram of the fine-grained extraction process provided by this application. First, Hough line detection and feature point extraction are performed on the local image. Hough line detection is mainly used to detect straight lines in an image. Its principle is to convert the line equation in the image into a point (polar diameter, polar angle) representation in Hough space (polar coordinate space), find the peak of the number of votes in Hough space, convert the peak back to image space, and finally determine the straight line in the image. Figure 13 Schematic diagram of Hough line detection provided for this application.

[0147] Figure 14 This is a schematic diagram of the straight line detection effect provided by this application. Figure 15 This is a schematic diagram of the feature point extraction effect provided by this application. Figure 14 As shown, obtain Line_1 and Line_2. The equations of these two lines in polar coordinates are as follows:

[0148] Line_1: x*cos(theta1)+y*sin(theta1)=ρ1;

[0149] Line_2: x*cos(theta2)+y*sin(theta2)=ρ2;

[0150] Where theta is the polar angle of the line, and ρ is the polar diameter of the line.

[0151] Image feature point extraction is mainly used to find feature points with unique properties in the image, such as corner points, edge intersections, spots, etc. These feature points can be accurately detected in different images and have a certain robustness to image changes (such as rotation, scaling, brightness changes, etc.). Commonly used feature point extraction algorithms include Harris corner detection algorithm, SIFT algorithm, SURF algorithm and FAST algorithm. As shown in the attached figure, the feature point position set {point1, point2} is obtained. This application uses Hough line detection and feature point extraction algorithm to obtain straight lines in local images.

[0152] Then, the corner point positions are determined based on the straight line and feature point information obtained above. Due to the complexity of the actual environment, Hough line detection and feature point extraction cannot produce stable results. Therefore, it is proposed to fuse the results of the two, as follows.

[0153] If there is no straight line in the local image and no feature points are extracted, it is considered that there is no corner point in the local image.

[0154] There are straight lines in the local image, and after line screening, there are two straight lines. Line screening is mainly based on judging the polar angle theta range of the line. The straight lines formed by the bricks are mainly horizontal or vertical lines with a certain inclination, and the horizontal and vertical straight lines are the most obvious image features in the local image. Set the polar angle range theta of the horizontal and vertical lines thresholdi , i = 1, 2, 3, 4, directly filter out the lines that are no longer within the set range in the Hough space.

[0155]

[0156] Furthermore, the intersection of the two lines is calculated by solving the system of equations of the two lines. The intersection point (x, y) is obtained by solving the system of equations:

[0157] x=(ρ2*sin(theta1)-ρ1*sin(theta2)) / sin(theta1-theta2);

[0158] y=(ρ1*cos(theta2)–ρ2*cos(theta1)) / sin(theta1-theta2).

[0159] Check whether the intersection point (x, y) is within the local image range. If the intersection point is within the image range, it is used as the corner point of the local image.

[0160] If the intersection point is not within the local image, or if only one line exists in the local image after line filtering, the algorithm further determines whether a feature point exists. If a feature point (point(x, y)) exists, the Euclidean distance (disToline) from all feature points to the line is calculated. The closest feature point to the line is searched for. If the distance from the line to the point is less than the set threshold (dis_thre), the point is considered a corner point for the local image. If no feature point exists, or the distance from the closest feature point to the line exceeds the set threshold (dis_thre), the local image is considered to have no corner points, or corner point extraction has failed.

[0161] Corner point filtering:

[0162] Corner point screening further processes the refined extracted corner points, eliminates erroneous corner points based on the positional relationship between adjacent corner points, and fills in the positions where no corner points are obtained in the local image.

[0163] 1. Based on each corner point Corner_i(x,y), search in the horizontal and vertical directions respectively to obtain the adjacent corner point set {Corner1, Corner2, ..., Corner_j}, j < 4.

[0164] 2. Calculate the distances between corner point Corner_i(x,y) and the corner points in the corner point set {disTocorner_i1, disTocorner_i2, ..., disTocorner_ij}. If disTocorner_ij is significantly longer than the other distances by about 2 times, it is considered that there is a missed corner point in that direction. The coordinates of the missed corner point are the center of Corner_i(x,y) and disTocorner_ij. If disTocorner_ij is significantly shorter than the distances in other directions, the corner point in that direction is considered to be an incorrectly detected corner point and is directly deleted.

[0165] 3. Traverse all corner points according to the above ideas, and finally retain only the square corner points as shown in the attached figure. Figure 16 Schematic diagram of the corner point coordinate distribution provided for this application.

[0166] Construct a coordinate table of the corner point image position and a coordinate table of the corresponding world coordinates.

[0167] The elements of the corner point image position coordinate table are the coordinates (u, v) of each corner point in the pixel coordinate system. The corresponding world coordinate corner point position requires manual measurement and input of the brick size. Here, the actual side length of the brick is set to 450mm. The first corner point in the lower left corner of the image is used as the origin to establish a 3D Cartesian coordinate system. Ignoring the Z direction, the world coordinate position of the corner point is as follows:

[0168] For example, the coordinate table of the corner point in the world coordinate system:

[0169]

[0170] Iteratively calculate camera parameters:

[0171] This step primarily utilizes Zhang Zhengyou's calibration method to solve for camera intrinsic and extrinsic parameters. The principle is briefly described. The camera calibration algorithm inputs a table of corner image position coordinates and their corresponding world coordinate coordinates to solve for and optimize intrinsic, extrinsic, and distortion parameters. Camera calibration is primarily based on the camera imaging model. Its essence is the transformation relationship between the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system. This transformation relationship is used to establish a model between the pixel coordinate system and the world coordinate system, ultimately optimizing the camera parameters. Figure 17 Schematic diagram of coordinate system conversion during the camera calibration process provided in this application.

[0172] World coordinate system: (Xw, Yw, Zw) is a user-defined coordinate system for the three-dimensional world. It is introduced to describe the position of the target object in the real world. Camera coordinate system: (Xc, Yc, Zc) is a coordinate system established on the camera. It is defined to describe the position of the object from the camera's perspective. The origin of the camera coordinate system is at the optical center of the camera, and the z-axis is parallel to the camera's optical axis. It serves as an intermediate link between the world coordinate system and the image / pixel coordinate system. Image coordinate system: (X, Y) is introduced to describe the projection and transmission relationship of the object from the camera coordinate system to the image coordinate system during the imaging process, making it easier to further obtain the coordinates in the pixel coordinate system. Pixel coordinate system: (u, v) is introduced to describe the coordinates of the image point on the digital image (photo) after the object is imaged. It is the coordinate system in which the information we actually read from the camera is located. Figure 18 The coordinate transformation relationship provided in this application.

[0173] like Figure 18 As shown in the figure, (u, v) are the coordinates of the corner point in the pixel coordinate system, (Xw, Yw) are the coordinates of the corner point in the world coordinate system, (u0, v0) are the coordinates of the image center origin in the pixel coordinate system, dx and dy are the physical dimensions of each pixel in the x and y directions of the image plane, respectively, and f is the camera focal length (the distance between the image plane and the origin of the camera coordinate system). The above are the camera intrinsic parameters. R is a 3×3 orthogonal rotation matrix, t is a three-dimensional translation vector, and the rotation and translation matrices are the camera extrinsic parameters.

[0174] The process of converting image coordinates to pixel coordinates introduces distortion parameters, including radial distortion parameters k1, k2, and k3, and tangential distortion parameters p1 and p2. The optimization goal is to minimize the reprojection error of corner points through continuous iteration. A random sample consensus algorithm (RANSAC) is used to obtain the optimal solution. The parameters of the mathematical model are estimated iteratively from a set of observed data, including outliers. Ultimately, the camera's intrinsic and extrinsic parameters, as well as the distortion coefficients, are obtained.

[0175] The RANSAC steps are as follows:

[0176] 1. Randomly select some corner points in the data and set them as the inner group (valid data).

[0177] 2. Calculate the model that fits the ingroup.

[0178] 3. Bring the other unselected points into the model just established for calculation to determine whether they are in the inner group.

[0179] 4. Note the number of in-groups.

[0180] 5. Repeat the above steps several times.

[0181] 6. Compare the number of inner groups in each calculation. The model established with the largest number is the solution.

[0182] Object detection and tracking.

[0183] First, pedestrian detection is performed on each frame in the real-time image sequence image1, image2, ..., imagen. Common target detection methods include the Faster RCNN series, the YOLO series, and the EfficientDet series. For pedestrian detection, algorithms such as human body detection, head and shoulder detection, and head detection can be used. Key point detection methods can also be combined to improve the accuracy of locating the head, feet, hands, and other parts of the human body. This paper uses YOLOv5's head detection as an example. After the image passes through the network, a series of head sets (box1, box2, box3, etc.) are output.

[0184] Then, for the set of heads obtained in the previous step, it is necessary to track them in real time, and use a multi-target tracking algorithm to generate a unique ID for all detection results. The more targets there are, the greater the computational complexity of the tracking algorithm, and the lower the efficiency and real-time performance of event recognition. To this end, this solution proposes to only track pedestrians who enter the vicinity of the detection line, and not track pedestrians who are farther away from the detection line. However, due to the influence of the camera focal length, the distance of the target pedestrian in the image is larger when it is closer and smaller when it is farther away. Therefore, it is proposed to judge whether the pedestrian is near the detection line based on the size of the pedestrian's head frame. The details are as follows:

[0185]

[0186] Among them, Dist i Target pedestrian obj i Distance sets the pixel distance of the detection line segment, w i Pedestrian obj output by the target detection algorithm i The head frame size (height or width) of the head frame, distance_headNum_threshold is the [Detection Area Distance] threshold set based on experience, which is used to adjust the size of the range near the detection line. The larger the value, the wider the detection range. Figure 19 Schematic diagram of pedestrians near the detection line provided for this application.

[0187] If Dist i ≥w i*distance_headNum_threshold, then the pedestrian obj i If the distance to the detection line is far and not near the detection line, target tracking will no longer be performed for the target. Otherwise, the target is considered to be in the area near the set detection line and target tracking is started. At present, the commonly used multi-target tracking algorithms can be simply divided into three categories: algorithms that do not use feature information matching (such as SORT and Bytetrack algorithms), algorithms that use feature information matching (such as deepsort algorithms), and algorithms that output feature information at the same time during detection (such as JDE and FairMOT algorithms). These algorithms are classified according to the matching method between different frames for the same target. Taking the deepSORT algorithm as an example, the present invention performs feature extraction on the target box obtained by detection, and then performs feature matching and status update. For the head sets box1, box2, box3, etc., the results obtained after tracking are {ID1, BOX1}, {ID2, BOX2}, {ID3, BOX3}, etc. Figure 20 Schematic diagram of pedestrian head detection provided for this application. Figure 20 The box in the figure is the head tracking output result.

[0188] Pedestrian speed calculation.

[0189] Pedestrian speed calculation is mainly based on speed = Distance / △T. Considering that Distance is affected by image focal length, distortion, etc., pedestrians moving faster in areas with large distortion and faster in areas close to the image, the speed calculation is inaccurate. Figure 21 The schematic diagram of the process of determining the pedestrian movement speed provided in this application includes obtaining the position coordinates and corresponding frame numbers of multiple targets, inputting the position frame of each target into the image correction module, calculating the actual movement distance Distance, compensating the actual movement distance based on the statistical model, and calculating the pedestrian's movement speed v.

[0190] Cache the detection and tracking results of M frames, then search for historical location coordinates that match multiple target IDs in M ​​frames and obtain the corresponding frame numbers. See the matching results in the figure. Figure 22 This is a schematic diagram of the positions of the detection frames for the same ID provided in this application.

[0191] The input of multiple target frames is fed into the image correction module. To improve the real-time performance of the algorithm, only the image data within the rectangular frame is corrected for lens distortion using the intrinsic parameters and distortion parameters obtained through camera calibration to generate the corrected image.

[0192] The coordinate relationship before and after radial distortion correction is as follows, where xp and yp are the positions of the ideal undistorted pixel points, and xrcorr and yrcorr are the positions of the pixel points after distortion.

[0193]

[0194] Tangential distortion requires two additional distortion parameters to describe. The coordinate relationship before and after correction is as follows, where xp and yp are the ideal undistorted pixel positions, and xrcorr and yrcorr are the distorted pixel positions.

[0195]

[0196] Calculate the actual movement distance based on the location boxes of multiple targets:

[0197] Based on the calibrated head frame image, the ranging model can use a pinhole imaging model or a complex 3D model. This section uses the pinhole model as an example. The relationship between the focal length F (unit pixel value), the width W of the target head (unit meter), the pixel width P of the target object in the image (unit pixel value), and the distance D of the target object from the camera (unit meter) is:

[0198] F=(P*D) / W.

[0199] Then, if F, P, and W are known, the distance D between the human body and the camera can be calculated:

[0200] D=(F*W) / P.

[0201] The target moving distance between two adjacent frames is:

[0202] distance=|Di-Di+1|.

[0203] As shown in the attached example, the actual distance traveled by the target pedestrian ID1 in M ​​frames is:

[0204] Distance_sum=distance23+distance34+distance45.

[0205] Finally, △T can be obtained according to the determined frame number, and the pedestrian's moving speed can be calculated:

[0206] speed=Distance_sum / △T.

[0207] Because there are still many estimation and calculation losses in the calibration of this application, and the target in the image is affected by focal length and occlusion, the target speed is larger when close up and smaller when far away, and the lateral movement speed is inaccurate. Based on the above model, the speed of a large number of pedestrians is statistically analyzed, and it is found that the speed is mainly related to the target's vertical coordinate in the image and the pedestrian's movement direction.

[0208] Figure 23This application provides a schematic diagram of the relationship between speed, the vertical coordinate of the target in the image, and the direction of movement of the pedestrian. Where speed is the pedestrian's speed, angle is the angle between the pedestrian's moving direction and the vertical direction of the image, called the moving direction angle, and y is the vertical coordinate of the center of the head frame.

[0209] Figure 24 The schematic diagram of the relationship between the speed and the ordinate of the target in the image provided in this application. Speed ​​and the ordinate are mainly in a linear relationship. The speed=Ay model is used to fit the above scatter plot to obtain the A parameter. The main principle of data fitting is to substitute the existing data into a selected fitting model, and to ensure the sum of the squared errors in an optimized way to determine the parameters to be determined in the model. It should be noted that the data fitting method given in this application is a polynomial model, which includes but is not limited to polynomial models (such as y=ax+b), natural number exponential models (such as y=e^x); power law models (such as y=x^a) and logarithmic models (such as y=log(x). There are also various least squares and gradient descent methods for model optimization solutions. The core of this method is to use the ordinate position and moving direction of the target in the image to correct the moving speed.

[0210] Figure 25 This is a schematic diagram of the relationship between speed and pedestrian movement direction provided by this application. Speed ​​is inversely related to the movement direction angle angle and is symmetrical about angle = 90. Use the speed = Csin(angle) model to fit the above scatter plot and obtain the parameter C.

[0211] By default, the walking speed of most pedestrians is similar within the movement range, and the speed value is corrected twice based on the initial model of the above two dimensions. The correction is as follows:

[0212] speed=(Distance_sum / △T) / Ay-Csin(angle);

[0213] speed=speed<0?0:speed.

[0214] After the above steps, the speed and movement direction of all target pedestrians can be calculated.

[0215] Event identification.

[0216] Based on the calculated passenger flow speed, the speed results are output at specified time intervals. If a group of people in the detection area exhibits noticeable fast or slow movement, with a speed greater than or less than a set threshold (no speed unit required), a clear image of the target is captured, the target is circled, and a video clip of the event is output. Alarms are generated based on the scenario, with a configurable interval for each alarm. Figure 26Schematic diagram of the subway scene test provided for this application. Figure 26 The corresponding text is "A young man running fast, speed: 1.7 m / s".

[0217] Figure 27 The schematic diagram of the passenger flow speed detection device provided in this application includes:

[0218] a correction module 271 for determining, in response to a passenger flow speed detection instruction, initial detection frames of different persons in a plurality of first images in a first surveillance video captured by an image acquisition device, and correcting the initial detection frames using a predetermined correction parameter of the image acquisition device to obtain corresponding target detection frames;

[0219] A first determining module 272 is configured to determine speed information of the different persons based on the timestamp information of the multiple first images and the position information of the target detection frames of the different persons;

[0220] The second determination module 273 is used to obtain multiple second images in a second surveillance video containing a floor tile area; determine the image position information of multiple corner points of the floor tile area in the second images for the multiple second images; and determine the correction parameters of the image acquisition device based on the image position information of the multiple corner points in the multiple second images and the pre-saved position information of the world coordinate system corresponding to the multiple corner points.

[0221] The second determination module 273 is used to perform personnel detection on the multiple second images, select the second image without personnel as the third image; identify the floor area sub-image in the multiple third images through the target detection algorithm, and determine the image position information of multiple corner points of the tile area sub-image.

[0222] The second determination module 273 is used to input the brick area sub-image into a trained corner detection network model, and determine multiple corner point areas in the brick area sub-image based on the corner detection network model; for the multiple corner point areas, perform straight line detection and feature point extraction on the corner point areas to determine the image position information of the corner points in the corner point areas.

[0223] The second determination module 273 is used to perform straight line detection on the corner point area. If there are two straight lines in the corner point area, the image position information of the corner point is determined according to the intersection of the two straight lines. If there is a straight line in the corner point area, feature point extraction is performed on the corner point area, and the image position information of the corner point is determined according to the feature point closest to the straight line. If there is no straight line in the corner point area, it is determined that there is no corner point in the corner point area.

[0224] The second determination module 273 is used to use the image position information of the intersection as the image position information of the corner point if the intersection of the two straight lines is within the corner point area; if the intersection of the two straight lines is not within the corner point area, extract feature points from the corner point area, and determine the image position information of the corner point based on the feature point closest to the two straight lines.

[0225] The second determination module 273 is configured to use the image position information of the feature point corresponding to the shortest distance from the feature point to the straight line as the image position information of the corner point if the shortest distance from the feature point to the straight line is less than a preset distance threshold; otherwise, determine that no corner point exists in the corner point area.

[0226] The second determination module 273 is further used to determine, with respect to the multiple corner points, multiple candidate corner points adjacent to the corner points; respectively determine the distances between the corner point and the multiple candidate corner points, and determine missed detection position information corresponding to the falsely detected corner points and the missed detection corner points based on the distances between the corner point and the multiple candidate corner points; supplement corner points in the missed detection position information, and delete the falsely detected corner points.

[0227] The correction module 271 is used to determine the initial head frames of different people in multiple first images in the first surveillance video captured by the image acquisition device; determine the distance between the initial head frame and a preset detection line, determine the initial head frame whose distance is less than a preset distance threshold as the initial detection frame, and filter out the initial head frame whose distance is not less than the preset distance threshold.

[0228] The first determination module 272 is also used to fit a first parameter based on the relationship between the speed information of the different people and the vertical coordinate of the target detection frame; fit a second parameter based on the relationship between the speed information of the different people and the moving direction angle; and update the speed information of the different people based on the first parameter and the second parameter.

[0229] The present application also provides an electronic device, such as Figure 28 As shown, it includes: a processor 281, a communication interface 282, a memory 283 and a communication bus 284, wherein the processor 281, the communication interface 282, and the memory 283 communicate with each other through the communication bus 284;

[0230] The memory 283 stores a computer program, and when the program is executed by the processor 281 , the processor 281 performs any of the above method steps.

[0231] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0232] The communication interface 282 is used for communication between the electronic device and other devices.

[0233] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk memory. Alternatively, the memory may be at least one storage device located away from the processor.

[0234] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0235] The present application also provides a computer storage readable storage medium, which stores a computer program that can be executed by an electronic device. When the program runs on the electronic device, the electronic device implements any of the above method steps when executing.

[0236] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0237] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A speed detection method, characterized in that: The method comprises: In response to a passenger flow speed detection instruction, determining initial detection frames of different persons in a plurality of first images in a first surveillance video captured by an image acquisition device, and correcting the initial detection frames using a predetermined correction parameter of the image acquisition device to obtain corresponding target detection frames; determining speed information of the different persons according to the timestamp information of the multiple first images and the position information of the target detection frames of the different persons; The process of determining the correction parameters of the image acquisition device includes acquiring a plurality of second images in a second surveillance video containing a floor tile area; determining image position information of a plurality of corner points of the floor tile area in the second images for the plurality of second images; and determining the correction parameters of the image acquisition device based on the image position information of the plurality of corner points in the plurality of second images and pre-stored position information of the world coordinate system corresponding to the plurality of corner points. Among them, corner points refer to the joint corners of adjacent floor tiles; After determining the image position information of the plurality of corner points of the floor tile area in the second image, the method further includes: For the multiple corner points, determine multiple candidate corner points adjacent to the corner points; Determining the distances between the corner point and the multiple candidate corner points respectively, and determining missed detection position information corresponding to the falsely detected corner points and the missed detection corner points according to the distances between the corner point and the multiple candidate corner points; Corner points are supplemented in the missed detection position information, and the falsely detected corner points are deleted.

2. The method according to claim 1, wherein For the plurality of second images, determining image position information of a plurality of corner points of the floor tile area in the second images includes: performing person detection on the plurality of second images, and selecting a second image in which no person is present as a third image; The floor tile region sub-images in the plurality of third images are identified by using a target detection algorithm, and image position information of a plurality of corner points of the floor tile region sub-images is determined.

3. The method according to claim 2, wherein Determining the image position information of multiple corner points of the floor tile area sub-image includes: Inputting the floor tile area sub-image into a trained corner detection network model, and determining a plurality of corner regions in the floor tile area sub-image based on the corner detection network model; For the multiple corner point regions, line detection and feature point extraction are performed on the corner point regions to determine image position information of the corner points in the corner point regions.

4. The method according to claim 3, wherein Performing line detection and feature point extraction on the corner point area to determine image position information of the corner points in the corner point area includes: Performing line detection on the corner point area, and if there are two straight lines in the corner point area, determining the image position information of the corner point according to the intersection of the two straight lines; If there is a straight line in the corner point area, feature point extraction is performed on the corner point area, and image position information of the corner point is determined based on the feature point closest to the straight line; If there is no straight line in the corner point area, it is determined that there is no corner point in the corner point area.

5. The method according to claim 4, wherein Determining the image position information of the corner point according to the intersection of the two straight lines includes: If the intersection of the two straight lines is within the corner point area, the image position information of the intersection is used as the image position information of the corner point; if the intersection of the two straight lines is not within the corner point area, feature points are extracted from the corner point area, and the image position information of the corner point is determined based on the feature point closest to the two straight lines.

6. The method according to claim 4, wherein Determining the image position information of the corner point according to the feature point closest to the straight line includes: If the shortest distance between the feature point and the straight line is less than a preset distance threshold, the image position information of the feature point corresponding to the shortest distance is used as the image position information of the corner point; otherwise, it is determined that no corner point exists in the corner point area.

7. The method according to claim 1, wherein Determining initial detection frames of different persons in a plurality of first images in a first surveillance video captured by the image capture device includes: Determine the initial head frames of different people in multiple first images in the first surveillance video captured by the image acquisition device; determine the distance between the initial head frame and a preset detection line, determine the initial head frame with a distance less than a preset distance threshold as the initial detection frame, and filter out the initial head frame with a distance not less than the preset distance threshold.

8. The method according to claim 1, wherein After determining the speed information of the different persons, the method further includes: Fitting a first parameter based on a relationship between the speed information of the different persons and the vertical coordinate of the target detection frame; According to the relationship between the speed information and the moving direction angles of the different persons, a second parameter is obtained by fitting; The speed information of the different persons is updated according to the first parameter and the second parameter.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 8 when executing a program stored in a memory.

Images