An edge management device with video data collection and analysis function

Abstract

The application relates to an edge management device with a video data collection and analysis function and belongs to the technical field of video data recognition and analysis. The device realizes tracking of a customer action route in selected continuous video frame images based on the characteristics that a human body corresponding connected domain has adjacent and relatively fixed positions, further analyzes customer waiting time and service processing time of different service windows, determines the service pressure of each service window according to the service processing time of each service window and the number of customers with to-be-handled services and the overall waiting time of customers with to-be-handled services, provides fine data support for subsequent service resource redistribution, and can effectively solve the technical problem that the video data collection and analysis function of the current edge management device is not fine enough in service pressure analysis, thereby causing low service resource utilization efficiency.

Description

Technical Field

[0001] This invention relates to the field of video data recognition and analysis technology, and in particular to an edge management device with video data acquisition and analysis functions. Background Technology

[0002] Edge management devices are integrated intelligent entities deployed at the edge of the sales hall, providing computing power and large-scale model support for the terminals. The video data acquisition and analysis function of edge management devices can dynamically adjust staff and equipment resources and optimize customer service experience by analyzing factors such as customer numbers and dwell time. Therefore, the video data acquisition and analysis function of edge management devices plays a crucial role in both operations and security management, and is an important functional component of edge management devices.

[0003] However, the video data acquisition and analysis functions in existing edge management devices can usually only count the number of people in the business hall as an indicator to assess customer flow. That is, they can only represent the overall business pressure of the business hall but cannot accurately determine the current business pressure of each service window. The subsequent realization of refined management of each window often still requires manual judgment to make service decisions and adjustments. The precision and real-time nature of the business pressure judgment are insufficient, which leads to the inability to maximize the utilization of service resources such as employees and equipment.

[0004] In other words, the edge management devices currently deployed at the edge of the business hall have insufficient fine-grained video data collection and analysis functions for analyzing business pressure, resulting in inefficient use of service resources. Summary of the Invention

[0005] In view of this, the present invention provides an edge management device with video data acquisition and analysis functions to solve the technical problem of low service resource utilization efficiency caused by the insufficient refinement of business pressure analysis in the video data acquisition and analysis functions of current edge management devices.

[0006] An edge management device with video data acquisition and analysis function according to the present invention includes:

[0007] The image acquisition module is used to acquire images of the detection area at a set interval to obtain a set of image frames for a set duration as the image set to be analyzed.

[0008] The customer trajectory recognition module is used to determine the connected components in each image frame of the image set to be analyzed based on the pixel color similarity of adjacent pixels in the image frame, to determine the similar connected components in the image set to be analyzed based on the color similarity and shape similarity of the connected components between any adjacent image frames in the image set to be analyzed, and to determine the position of the same customer in each image frame based on the similar connected components in the image set to be analyzed.

[0009] The business window status determination module is used to determine the current waiting time of each customer and the average service time of each business window based on the position of the same customer in each image frame. Based on the current waiting time of each customer, the current average service time of the current business window, and the number of customers who have not yet processed their business at the current business window, it determines the remaining processing time of the current business window and the total waiting time of customers who have not yet processed their business at the current business window.

[0010] The window service pressure determination module is used to determine the service pressure of each service window based on the remaining processing time for each service window and the total waiting time of customers who have not yet completed their business at each service window.

[0011] Furthermore, a correction interval is obtained by adjusting the set interval according to the degree of difference between the image frames at any adjacent time in the image frames of a set duration. The correction interval is inversely proportional to the degree of difference between the image frames.

[0012] Image frames are selected from the set of image frames with a set time interval at a modified interval, and the selected set of image frames is used as the image set to be analyzed.

[0013] Furthermore, the step of adjusting the set interval based on the degree of difference in image frames at any adjacent time points within an image frame of a set duration to obtain a correction interval includes:

[0014] All image frames in the image frame of a set duration are converted to grayscale to obtain grayscale image frames. The average value of the grayscale value difference of corresponding pixels between two adjacent grayscale image frames is calculated as the degree of grayscale difference between the two adjacent grayscale image frames.

[0015] Select the maximum value of the grayscale difference between any two adjacent grayscale image frames in all grayscale image frames, and construct a correction interval that is inversely proportional to the maximum value of the grayscale difference and is not less than the lower limit of the preset correction interval and not greater than the upper limit of the preset correction interval.

[0016] Furthermore, the step of determining the connected components within each image frame of the image set to be analyzed based on the pixel color similarity of adjacent pixels within the image frame includes:

[0017] Calculate the absolute value of the difference between each color channel value between any two adjacent pixels in the current image frame, construct a pixel color similarity that is inversely proportional to the sum of the absolute values ​​of the differences between all color channel values, and classify two adjacent pixels whose pixel color similarity is greater than a preset pixel color similarity threshold into the same connected component.

[0018] Further, determining the connected component color similarity between any two adjacent image frames in the image set to be analyzed includes:

[0019] Calculate the average channel value of each pixel in the current connected component under the current color channel and use it as the color feature value of the current connected component under the current color channel;

[0020] In any two adjacent image frames, select one connected component and use the two selected connected components as the previous and next contrast connected components, respectively. Calculate the absolute value of the difference between the color feature values ​​of the previous and next contrast connected components in each color channel, and construct the connected component color similarity that is inversely proportional to the sum of the absolute values ​​of the differences of all color feature values.

[0021] Further, determining the similarity of connected component shapes between any adjacent image frames in the image set to be analyzed includes:

[0022] In any two adjacent image frames, select one connected component respectively, take the connected component with fewer edge pixels as the reference connected component, and take the other connected component as the comparison connected component.

[0023] Starting from a preset edge pixel on the reference connected domain, calculate the orientation of the next edge pixel relative to the previous edge pixel in the set direction, and use the preset code corresponding to the orientation as the code of the next edge pixel, until all edge pixels on the reference connected domain are traversed to generate a reference code chain.

[0024] Starting from any edge pixel in the connected component, the orientation of the next edge pixel relative to the previous edge pixel is calculated in sequence according to the set direction, and the preset code corresponding to the orientation is used as the code of the next edge pixel. This process continues until the number of encoded edge pixels reaches the total number of edge pixels in the reference connected component, and then a matching coding chain corresponding to the aforementioned edge pixel is generated.

[0025] Calculate the coding chain similarity between the reference coding chain and each aligned coding chain, and use the maximum value of the coding chain similarity as the connected component shape similarity.

[0026] Furthermore, the similarity of the encoding chains is:

[0027] ,

[0028] in, This represents the coding chain similarity between the reference coding chain calculated based on the w-th connected component in the i-th video image and the u-th connected component in the (i+1)-th video image, and the y-th alignment coding chain. This represents the number of edge pixels in the connected component with fewer edge pixels between the w-th and u-th connected components. This represents the j-th encoded value in the coding chain corresponding to the w-th connected component in the i-th video image, when calculating the coding chain similarity between the reference coding chain and the y-th comparison coding chain based on the w-th connected component in the i-th video image and the u-th connected component in the (i+1)-th video image. This represents the j-th encoded value in the coding chain corresponding to the u-th connected component in the (i+1)-th video image when calculating the coding chain similarity between the reference coding chain and the y-th comparison coding chain based on the w-th connected component in the i-th video image and the u-th connected component in the (i+1)-th video image. The parameters are used to ensure that the fraction is meaningful.

[0029] Furthermore, determining the location of the same customer in each image frame based on similar connected components in the image set to be analyzed includes:

[0030] Similar connected components in the image set to be analyzed are marked with the same number, while non-similar connected components are marked with different numbers. The numbers corresponding to any two adjacent connected components in each image frame are used to form a number set. For any kind of similar connected component among all kinds of similar connected components, the number set with the highest frequency among all number sets containing the number corresponding to the number of ...

[0031] The recombined number sets containing duplicate numbers from all similar connected component sets are merged into a fused number set. The recombined number sets containing numbers not duplicated from other recombined number sets are directly used as the fused number set. The connected components corresponding to numbers in any fused number set in the current image frame are combined to form a connected component group for that fused number set in the current image frame. The position of the connected component group of any fused number set in all image frames is used as the position of the same object in each image frame. Human-shaped objects are identified from the objects whose positions in each image frame are determined, thus completing the determination of the same customer's position in each image frame.

[0032] Furthermore, determining the current waiting time for each customer and the average service time for each service window includes:

[0033] The time a customer spends in the waiting area is taken as the customer's waiting time, and the average time each customer spends at the current service window is taken as the average service time for the current service window.

[0034] Furthermore, determining the remaining processing time at the current service window and the total waiting time for customers who have not yet had their business processed at the current service window includes:

[0035] The remaining processing time at the current service window is determined based on the average service time at the current service window and the number of unregistered customers at the current service window.

[0036] Calculate the ratio of the number of customers who have not registered at the current service window to the total number of all customers, and use the sum of the ratio and the time spent by all customers in the waiting area as the total waiting time for customers who have not registered at the current service window.

[0037] The advantages of this invention compared to the prior art are:

[0038] This invention leverages the relatively fixed adjacent and relative positions of connected domains corresponding to the human body to track customer movement routes within selected consecutive video frames. It then analyzes customer waiting times and processing times for different service windows. Based on the processing time of each service window, the number of customers with pending tasks, and the overall waiting time of those customers, the invention determines the workload of each service window. This provides refined data support for the subsequent reallocation of service resources and effectively addresses the technical problem of insufficient refinement in the analysis of service workload by the current edge management devices' video data acquisition and analysis functions, leading to low service resource utilization efficiency. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a block diagram of the module structure of an edge management device with video data acquisition and analysis function provided in Embodiment 1 of the present invention;

[0041] Figure 2 This is a schematic diagram of the encoding corresponding to each orientation of a subsequent edge pixel relative to the preceding edge pixel in an adjacent edge pixel of a connected domain, provided in Embodiment 1 of the present invention. Detailed Implementation

[0042] To further illustrate the technical solution of the present invention, specific embodiments are described below.

[0043] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a particular feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. Furthermore, a particular feature, structure, or characteristic in one or more embodiments may be combined in any suitable form, and the terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless otherwise specifically emphasized.

[0044] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0045] See Figure 1 This is a block diagram of the module structure of an edge management device with video data acquisition and analysis function provided in Embodiment 1 of the present invention, as shown below. Figure 1 As shown, the device includes an image acquisition module 11, a customer trajectory recognition module 12, a business window status determination module 13, and a window business pressure determination module 14. The specific functions of each module are described below:

[0046] The image acquisition module 11 is used to acquire images of the detection area at a set interval to obtain a set of image frames for a set duration as the image set to be analyzed.

[0047] The detection area, i.e., the business hall, is captured by cameras at set intervals to obtain continuous image frames. Since changes in video images within the business hall are often caused by the movement of people, and the speed of human movement is usually relatively slow, one video frame is acquired every 1 second, i.e., the set interval is 1 second. Based on the average speed of business processing within the business hall and the patience of customers, the approximate duration of each customer's business can be determined. This allows for the determination of the set duration of the video to be acquired for subsequent analysis. In this embodiment, the set duration is preferably 1 hour, thus obtaining a set of image frames over one hour, which is recorded as the image set to be analyzed.

[0048] The customer trajectory recognition module 12 is used to determine the connected components in each image frame of the image set to be analyzed based on the pixel color similarity of adjacent pixels in the image frame, to determine the similar connected components in the image set to be analyzed based on the color similarity and shape similarity of the connected components between any adjacent image frames in the image set to be analyzed, and to determine the position of the same customer in each image frame based on the similar connected components in the image set to be analyzed.

[0049] Because different service windows handle different types of services, their processing times will vary. When serving customers, it's necessary to take appropriate service measures based on the customer's already waited time and the expected wait time to avoid service backlog and inefficiency. Furthermore, the processing time for each service window may vary depending on factors such as the speed of the staff and the type of service. In other words, the processing time for each window is not a fixed value and is not always strictly the same. Therefore, it's necessary to analyze the average processing time for each window based on customer movement routes to estimate the customer's queuing time. This requires determining the position of each customer in each image frame within the image set to determine their movement trajectory.

[0050] Since the image region corresponding to each customer in an image frame is generally composed of blocks made up of several similar pixels, that is, the customer image region can be regarded as composed of several connected components, this embodiment first determines the connected components in each image frame of the image set to be analyzed based on the pixel color similarity of adjacent pixels within a single image frame, including:

[0051] Calculate the absolute value of the difference between each color channel value between any two adjacent pixels in the current image frame, construct a pixel color similarity that is inversely proportional to the sum of the absolute values ​​of the differences between all color channel values, and classify two adjacent pixels whose pixel color similarity is greater than a preset pixel color similarity threshold into the same connected component.

[0052] The formulaic representation of the pixel color similarity between adjacent pixels in the image frame calculated during the above connected component determination process is as follows:

[0053] ,

[0054] in, This represents the pixel color similarity between the j-th pixel and the q-th pixel adjacent to the j-th pixel in the current image frame. This indicates the total number of color channels (here, we use the RGB three-channel value, so...). ), This represents the color channel value of the j-th pixel in the current image frame (where y = 1, 2, 3, corresponding to the R channel, G channel, and B channel, respectively). This represents the color channel value of the y-th pixel in the current image frame. This represents the absolute value of the difference between the y-th color channel values ​​of adjacent pixels. The smaller the absolute value, the closer the colors of the two pixels are. To ensure the parameters are meaningful, and to guarantee that the fraction is meaningful, the following is specified here: . This represents the normalization function.

[0055] Because the color of different parts of a person's body usually does not change in different images, and multiple connected components corresponding to the same person in different images are always arranged together in a relatively fixed relative positional relationship, multiple connected components belonging to the same person or object can be identified by judging the positional arrangement invariance of connected components with high similarity between different image frames in the image set to be analyzed, thus completing the identification of customer trajectories in the image set to be analyzed.

[0056] Therefore, it is necessary to first identify the connected components with high similarity among all connected components of all image frames in the image set to be analyzed, which are likely to be the same image region. This embodiment selects to combine color similarity and shape similarity between connected components to complete the connected component similarity calculation:

[0057] First, determine the connected component color similarity between any two adjacent image frames in the image set to be analyzed, including:

[0058] Calculate the average channel value of each pixel in the current connected component under the current color channel and use it as the color feature value of the current connected component under the current color channel;

[0059] In any two adjacent image frames, select one connected component and use the two selected connected components as the previous and next contrast connected components, respectively. Calculate the absolute value of the difference between the color feature values ​​of the previous and next contrast connected components in each color channel, and construct the connected component color similarity that is inversely proportional to the sum of the absolute values ​​of the differences of all color feature values.

[0060] Its formulaic manifestation process is as follows:

[0061] First, the average y-channel value of all pixels in the w-th connected component of the i-th image frame is used as the y-channel color feature value of the w-th connected component in the i-th image frame. Then, the color similarity between connected components in adjacent images is calculated.

[0062] ,

[0063] in, This represents the color similarity between the w-th connected component in the i-th image frame and the u-th connected component in the (i+1)-th image frame. This indicates the total number of color channels (here, we use the RGB three-channel value, so...). ), This represents the y-th color feature value of the w-th connected component in the i-th image frame. This represents the y-th color feature value of the u-th connected component in the (i+1)-th image frame. To ensure the parameters are meaningful, and to guarantee that the fraction is meaningful, the following is specified here: .

[0064] Then, the similarity of connected component shapes between any two adjacent image frames in the image set to be analyzed is determined, including:

[0065] In any two adjacent image frames, select one connected component respectively, take the connected component with fewer edge pixels as the reference connected component, and take the other connected component as the comparison connected component.

[0066] Starting from a preset edge pixel on the reference connected domain, calculate the orientation of the next edge pixel relative to the previous edge pixel in the set direction, and use the preset code corresponding to the orientation as the code of the next edge pixel, until all edge pixels on the reference connected domain are traversed to generate a reference code chain.

[0067] Starting from any edge pixel in the connected component, the orientation of the next edge pixel relative to the previous edge pixel is calculated in sequence according to the set direction, and the preset code corresponding to the orientation is used as the code of the next edge pixel. This process continues until the number of encoded edge pixels reaches the total number of edge pixels in the reference connected component, and then a matching coding chain corresponding to the aforementioned edge pixel is generated.

[0068] Calculate the coding chain similarity between the reference coding chain and each aligned coding chain, and use the maximum value of the coding chain similarity as the connected component shape similarity.

[0069] Specifically, the direction setting refers to either clockwise or counterclockwise. It also involves calculating the orientation of the subsequent edge pixel relative to the previous edge pixel and using a preset code corresponding to that orientation as the encoding of the subsequent edge pixel. The implementation method is described in [reference needed]. Figure 2 First, the orientation of the next edge pixel relative to the previous edge pixel is preset to a set number of orientations (e.g., ...). Figure 2 As shown, this embodiment preferably has 8 (preferably), and assigns a corresponding encoding value to each preset orientation, then as follows: Figure 2 As shown, assuming the orientation of the subsequent edge pixel relative to the previous edge pixel is horizontal to the right, then the encoding value of the subsequent edge pixel is 0. Thus, the reference encoding chain and the alignment encoding chain with the same number of edge pixels as the aligned connected region can be obtained.

[0070] Then, the encoding chain similarity between the reference encoding chain and each aligned encoding chain is calculated:

[0071] ,

[0072] in, This represents the coding chain similarity between the reference coding chain calculated based on the w-th connected component in the i-th video image and the u-th connected component in the (i+1)-th video image, and the y-th alignment coding chain. This represents the number of edge pixels in the connected component with fewer edge pixels between the w-th and u-th connected components. This represents the j-th encoded value in the coding chain corresponding to the w-th connected component in the i-th video image, when calculating the coding chain similarity between the reference coding chain and the y-th comparison coding chain based on the w-th connected component in the i-th video image and the u-th connected component in the (i+1)-th video image. This represents the j-th encoded value in the coding chain corresponding to the u-th connected component in the (i+1)-th video image when calculating the coding chain similarity between the reference coding chain and the y-th comparison coding chain based on the w-th connected component in the i-th video image and the u-th connected component in the (i+1)-th video image. The parameters are used to ensure that the fraction is meaningful.

[0073] It should be noted that the above parameters as well as This refers to the encoded values ​​in the encoding chain. Since the starting point of the encoding chain can be any of the edge pixels of the connected component, meaning that the starting point of the encoding chain is often not the starting point of an edge pixel, the above parameters... as well as The sequence number of the encoded value referred to is not the same as the sequence number of the edge pixel of the connected domain; it specifically refers to the sequence number of the encoded value in the encoding chain.

[0074] Furthermore, it is easy to understand that the upper limit of the index y of the matching coding chain is equal to the total number of pixels on the upper edge of the matched connected region. Thus, the same number of coding chain similarities as the total number of pixels on the upper edge of the matched connected region can be obtained.

[0075] The maximum value of the encoding chain similarity is used as the similarity of the connected component shape between any two adjacent image frames in the image set to be analyzed:

[0076] ,

[0077] in, This represents the similarity in shape between the w-th connected component in the i-th video image and the u-th connected component in the (i+1)-th video image. This represents the maximum value function, i.e., all the functions obtained above. Take the maximum value from the middle.

[0078] The overall similarity between two connected components is calculated by combining the similarity of their shapes and colors.

[0079] ,

[0080] in, This represents the combined similarity between the w-th connected component in the i-th video image and the u-th connected component in the (i+1)-th video image. This represents the normalization function.

[0081] Among them, two connected components with a comprehensive similarity greater than a comprehensive similarity threshold are considered to be the same connected component in suspected different images. Here, it is defined that when... When the overall similarity threshold is 0.8, the two connected components are considered to be similar connected components.

[0082] Then, the location of the same customer in each image frame is determined based on similar connected components in the image set to be analyzed, including:

[0083] Similar connected components in the image set to be analyzed are marked with the same number, while non-similar connected components are marked with different numbers. The numbers corresponding to any two adjacent connected components in each image frame are used to form a number set. For any kind of similar connected component among all kinds of similar connected components, the number set with the highest frequency among all number sets containing the number corresponding to the number of ...

[0084] The recombined number sets containing duplicate numbers from all similar connected component sets are merged into a fused number set. The recombined number sets containing numbers not duplicated from other recombined number sets are directly used as the fused number set. The connected components corresponding to numbers in any fused number set in the current image frame are combined to form a connected component group for that fused number set in the current image frame. The position of the connected component group of any fused number set in all image frames is used as the position of the same object in each image frame. Human-shaped objects are identified from the objects whose positions in each image frame are determined, thus completing the determination of the same customer's position in each image frame.

[0085] Specifically, firstly, based on the aforementioned determined similar connected components, all connected components in the image set to be analyzed that are identified as similar need to be labeled with the same number, while connected components identified as different connected components are not labeled with the same number. In this embodiment, it is preferable to label the similar connected components between adjacent image frames sequentially according to time order, until all image frames have been traversed, completing the labeling of all similar and dissimilar connected components in the image set to be analyzed.

[0086] For example, taking a connected component in the i-th image frame as a reference connected component, in the (i+1)-th image frame, there will be a similar connected component whose overall similarity value with the reference connected component is greater than the overall similarity threshold. Therefore, the reference connected component and its corresponding similar connected component are both labeled with the same number, such as number 'a'. Simultaneously, in the i-th image, there may also be other connected components besides the reference connected component whose overall similarity value with the connected component numbered 'a' in the (i+1)-th image frame is also greater than the overall similarity threshold. In this case, such connected components in the i-th image are also labeled with number 'a'. That is, if connected component A in the i-th image frame is identified as a similar connected component to connected components C and D in the (i+1)-th image frame, they are both initially labeled with number 'a'. Subsequently, if connected component B in the i-th image frame is also similar to connected components C and D in the (i+1)-th image frame, then connected component B is also labeled with number 'a'.

[0087] Then, based on the number of each connected component in the set of images to be analyzed, the position of each client in each image frame is determined.

[0088] Specifically, the connected components corresponding to the human body will always remain adjacent in different images, and the color and shape of the connected components will remain basically unchanged throughout all image frames, so that their corresponding numbers will continue to appear for a set period of time. However, a significant portion of the connected components in the environment will be segmented or occluded in different forms at different times due to the movement of people. Therefore, the numbers corresponding to the connected components in the environment usually cannot continue to appear for a set period of time. For example, a certain ground area in the environment may not be occluded by people for a period of time, forming a connected component of a corresponding shape. However, at another time, it may be segmented by people or objects, forming two new connected components of different shapes. Because the shape has changed, even if the connected components before and after the occlusion segmentation have a very high degree of color similarity, they will not be considered similar connected components and will not be marked with the same number.

[0089] Therefore, based on the above characteristics, this embodiment, according to the numbering of each connected component in the image set to be analyzed, constructs a number set by assigning numbers to any two adjacent connected components in each image frame. Within the range of the image set to be analyzed, it identifies all number sets containing numbers corresponding to a certain type of similar connected component. The number set with the highest frequency among the identified number sets is taken as the target number set for that type of similar connected component. Since the color and shape of the connected components in the personnel area are more stable than those in the environment area, by filtering by the highest frequency, number sets formed by adjacent personnel and environment connected components can be excluded, ensuring that the connected components corresponding to the numbers in the target number set are either purely personnel or environment connected components.

[0090] However, as described above regarding the characteristics of connected components in a human body region, if a similar connected component is located in the central part of the human body, the frequency of the number set formed by it and other different connected components in the surrounding human body region will be equal to the frequency of the human body in all image frames. This results in multiple target number sets for any kind of similar connected component. Since the connected components corresponding to the target number sets all belong to the same type of region (both are human body regions or both are environmental regions), it is necessary to combine the numbers in the target number sets of any kind of similar connected component to form a recombined number set corresponding to any kind of similar connected component.

[0091] Then, since the number of connected components contained in the personnel area or the environment area is generally large, after obtaining the recombined number set of any of the above-mentioned similar connected components, a single repeated number set cannot completely represent the personnel area (or a certain environment area) in most cases. Therefore, in this embodiment, the repeated code set with repeated numbers in the recombined number sets corresponding to all kinds of similar connected components is merged again to obtain a merged number set; and, for the recombined number sets corresponding to all kinds of similar connected components, the repeated number set that does not have repeated numbers with other recombined number sets is directly used as the merged number set.

[0092] After obtaining the set of fused numbers, each set of fused numbers corresponds to the total set of numbers of each connected component under a personnel area or environmental area. Then, by combining the connected components corresponding to the numbers in any set of fused numbers in the current image frame to form a connected component group of any set of fused numbers in the current image frame, the position of the connected component group of any set of fused numbers in all image frames can be used as the position of the same object in each image frame. That is, the position or trajectory of the object corresponding to any set of fused numbers in each image frame can be determined. By identifying human-shaped objects from the objects whose positions in each image frame have been determined, the position of the same customer in each image frame can be determined, and finally the movement trajectory of each customer in the detection area or business hall can be determined.

[0093] The process of identifying human-shaped objects from those already located in each image frame is achieved by a trained CNN convolutional neural network that identifies objects and filters out those that are classified as human-shaped.

[0094] The business window status determination module 13 is used to determine the current waiting time of each customer and the average service time of each business window based on the position of the same customer in each image frame. Based on the current waiting time of each customer, the average service time of the current business window, and the number of customers who have not yet processed their business at the current business window, it determines the remaining processing time of the current business window and the total waiting time of customers who have not yet processed their business at the current business window.

[0095] Since the locations of the service areas and waiting areas within the business hall are fixed, in order to determine whether a customer is waiting or conducting business, the areas of each service window in the image frame are pre-divided, and the areas outside each service window area are designated as waiting areas.

[0096] When a customer is in the service area of ​​a specific service window, they are considered to be processing their business. When a customer is outside the service area, they are considered to be waiting in a queue. This allows us to obtain the waiting time for each customer within a set time period. Because waiting times may exceed the set time of 1 hour when there are many people in the queue or the business is complex, if a customer is in a non-service area for the entire set time corresponding to the current image set being analyzed, their waiting time for that customer within the current set time period is recorded. This waiting time must be added to the waiting time for the next set time period. Thus, by accumulating the waiting times for each customer after they appear within the service hall area, we obtain the total waiting time for each customer. Until the customer enters the service area.

[0097] The average service time for each customer at each service window is determined based on the average time each customer spends at that service window area. This refers to the time from when the customer enters the service window area until they leave. Similarly, if a customer does not leave the service window area within a certain set time period, the time the customer spends within that time period is recorded and added to the analysis for the next set time period. Therefore, the average service time for each service window is determined based on the average time each customer spends at the current service window. .

[0098] Since customers need to register the type of service they wish to receive and obtain a number before processing their business, and different service windows correspond to different service types, we can calculate the remaining processing time at the current service window and the total waiting time for customers not yet having their business processed at the current service window, based on the total number of registered customers for each service type and the average service time of the corresponding service window. This includes:

[0099] The remaining processing time at the current service window is determined based on the average service time at the current service window and the number of unregistered customers at the current service window.

[0100] Calculate the ratio of the number of customers who have not registered at the current service window to the total number of all customers, and use the sum of the ratio and the time spent by all customers in the waiting area as the total waiting time for customers who have not registered at the current service window.

[0101] Specifically, the formulaic representation of the remaining processing time at the current service window and the total waiting time for customers who have not yet completed their business at the current service window is as follows:

[0102] ,

[0103] in, This indicates the remaining processing time required for the a-th service window. This represents the average service time of the a-th service window. This represents the number of customers who have not registered for the corresponding business type at window a.

[0104] Since the above only obtained the total waiting time of all customers based on the waiting time of each customer within the service hall, but does not know which customer is waiting for which service window, the total waiting time of customers at each service window is calculated based on the total total waiting time of all customers in the service hall, combined with the proportion of the number of unregistered customers for the corresponding service type at each service window to the total number of customers waiting for service:

[0105] ,

[0106] in, This indicates the total waiting time for customers who have not yet completed their business at the a-th service window. This indicates the total number of customers currently in a waiting state. This represents the current total waiting time for the k-th customer.

[0107] The window service pressure determination module 14 is used to determine the service pressure of each service window based on the remaining processing time of each service window and the total waiting time of customers who have not yet processed their business at each service window.

[0108] After obtaining the remaining processing time for each service window and the total waiting time for customers who have not yet completed their business, the workload of each service window can be determined by combining these two data points.

[0109] ,

[0110] in, This indicates the workload of the a-th business window. This represents the normalization function.

[0111] By assessing the workload of each service window, the relative workload of each window can be precisely determined, providing more detailed data support for subsequent reallocation of service resources and enabling more efficient and rational service delivery. The specific reallocation of service resources can be flexibly configured by operators based on actual circumstances. For example, this could include:

[0112] when When the preset business pressure threshold is reached, it is considered that the current window has a large customer flow. For windows with a large customer flow, more windows for that business type can be opened as appropriate.

[0113] Customers are prioritized based on their wait time.

[0114] For business types that can be completed at self-service machines, customers with longer waiting times will be given priority to use the self-service machines.

[0115] For customers who cannot conduct business at self-service machines, if it is not possible to add more windows, personalized product recommendations will be made based on the type of business they are conducting, or the waiting time for specific business will be explained.

[0116] The advantages of the embodiments of the present invention compared with the prior art are as follows:

[0117] (1) Analyze the movement route of each customer, obtain the waiting time of each customer and the average business processing time of each window, and then analyze the business pressure of each business processing window. This provides data support for subsequent targeted and refined service adjustment measures, which can improve the efficiency of business processing resource utilization and enhance user experience.

[0118] (2) Based on the similarity of connected components in adjacent images, similar connected components are determined, and the frequency of their combination in adjacent images is used to identify the same person in different images. This allows for accurate tracking of customer movement routes, which in turn reflects the pressure on each business window based on the customer movement routes, thereby improving the accuracy of the analysis results on the business pressure of each business window in the current business hall.

[0119] Embodiment 2 of the present invention also provides an edge management device with video data acquisition and analysis functions. The device includes the modules and functions described in Embodiment 1 above, and the image acquisition module to be analyzed further includes the following functions:

[0120] This invention determines customer waiting time and service processing time based on customer movement routes. Therefore, video frames where the customer has not moved do not require high-frequency acquisition for determining the customer's movement route or location at different times. Thus, this embodiment preferably improves analysis efficiency by reselecting image frames acquired at set intervals for a set duration, removing unnecessary frames.

[0121] First, the correction interval is obtained by adjusting the set interval according to the degree of difference between the image frames at any adjacent time in the image frames of a set duration. The correction interval is inversely proportional to the degree of difference between the image frames.

[0122] Image frames are selected from the set of image frames with a set time interval at a modified interval, and the selected set of image frames is used as the image set to be analyzed.

[0123] The correction interval is obtained by adjusting the set interval based on the degree of difference between image frames at any adjacent time points within an image frame of a set duration, including:

[0124] All image frames in the image frame of a set duration are converted to grayscale to obtain grayscale image frames. The average value of the grayscale value difference of corresponding pixels between two adjacent grayscale image frames is calculated as the degree of grayscale difference between the two adjacent grayscale image frames.

[0125] Select the maximum value of the grayscale difference between any two adjacent grayscale image frames in all grayscale image frames, and construct a correction interval that is inversely proportional to the maximum value of the grayscale difference and is not less than the lower limit of the preset correction interval and not greater than the upper limit of the preset correction interval.

[0126] Specifically, considering that changes in video images within the service hall (i.e., the detection area) are mainly caused by personnel activity, a large degree of change in consecutive video frames indicates frequent personnel activity within the service hall, thus requiring more frequent video frame images. Conversely, a smaller degree of change in consecutive video frames allows for a more appropriate increase in the video frame interval. Therefore, the degree of difference between any two adjacent image frames within the set of image frames for a given duration is first calculated based on the overall difference between adjacent image frames.

[0127] Each image frame in the obtained image frame set for the set duration is converted to grayscale, and the degree of difference in grayscale values ​​of pixels in adjacent grayscale image frames is calculated:

[0128] ,

[0129] in, This represents the degree of difference in grayscale values ​​between pixels in the i-th grayscale image frame and the (i+1)-th grayscale image frame. This indicates the number of pixels contained in an image frame. This represents the grayscale value of the j-th pixel in the i-th grayscale image frame. This represents the grayscale value of the j-th pixel in the (i+1)-th grayscale image frame. This represents the absolute value of the difference in grayscale values ​​between corresponding pixels in two adjacent video images. The larger the absolute value, the greater the difference between the adjacent images, indicating more frequent human activity. This represents the normalization function.

[0130] Therefore, the grayscale difference between any two adjacent grayscale image frames is calculated, which is equivalent to obtaining the grayscale difference of pixels in all adjacent image frames within the image frame set for a given duration. Then, the maximum value of the grayscale difference between any two adjacent grayscale image frames is selected from all grayscale image frames.

[0131] ,

[0132] in, This represents the maximum value of the difference in image quality between any two adjacent frames within a set duration. This indicates the total number of images contained in an image frame. The maximum value function is used here because, due to the influence of the type of business handled by customers, there may be some locations with dense customer traffic and some locations with fewer customers. In order to more accurately obtain the degree of difference between adjacent video images in locations with dense customer traffic, the maximum value is used here.

[0133] The greater the difference between adjacent images, the smaller the interval should be when selecting video images. Therefore, based on the initial set interval, the correction interval is calculated:

[0134] ,

[0135] in, Indicates the correction interval. This indicates the initial set interval (here) ), The greater the difference in image frames at any adjacent moments within a given duration, the smaller the corresponding correction interval. To ensure the parameters are valid and to guarantee the rationale is meaningful, the following settings are used: . This indicates rounding up. Because the interval between video frames cannot be infinitely small or large, it is defined here based on the normal frequency of human activity. The minimum value is 1 second, and the maximum value is 30 seconds.

[0136] Therefore, with To correct the interval, image frames are selected from the set of image frames for a set duration. The selected set of image frames is used as the image set to be analyzed. This can reduce the amount of data to be processed and increase the efficiency of analysis while ensuring the accuracy of subsequent analysis.

[0137] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An edge management device with video data acquisition and analysis functions, characterized in that, The device includes: The image acquisition module is used to acquire images of the detection area at a set interval to obtain a set of image frames for a set duration as the image set to be analyzed. The customer trajectory recognition module is used to determine the connected components in each image frame of the image set to be analyzed based on the pixel color similarity of adjacent pixels in the image frame, to determine the similar connected components in the image set to be analyzed based on the color similarity and shape similarity of the connected components between any adjacent image frames in the image set to be analyzed, and to determine the position of the same customer in each image frame based on the similar connected components in the image set to be analyzed. The business window status determination module is used to determine the current waiting time of each customer and the average service time of each business window based on the position of the same customer in each image frame. Based on the current waiting time of each customer, the current average service time of the current business window, and the number of customers who have not yet processed their business at the current business window, it determines the remaining processing time of the current business window and the total waiting time of customers who have not yet processed their business at the current business window. The window service pressure determination module is used to determine the service pressure of each service window based on the remaining processing time of each service window and the total waiting time of customers who have not yet processed their business at each service window. The correction interval is obtained by adjusting the set interval according to the degree of difference between the image frames at any adjacent time in the image frames of a set duration. The correction interval is inversely proportional to the degree of difference between the image frames. Image frames are selected from the set of image frames with a set time interval at a correction interval, and the selected set of image frames is used as the set of images to be analyzed. Determining the current waiting time for each customer and the average service time for each service window includes: The time a customer spends in the waiting area is taken as the current customer's waiting time, and the average time each customer spends at the current service window is taken as the average service time of the current service window. The determination of the remaining processing time at the current service window and the total waiting time for customers who have not yet had their business processed at the current service window includes: The remaining processing time at the current service window is determined based on the average service time at the current service window and the number of unregistered customers at the current service window. Calculate the ratio of the number of customers who have not registered at the current service window to the total number of all customers, and multiply the ratio by the sum of the time all customers spend in the waiting area as the total waiting time for customers who have not registered at the current service window.

2. The edge management device with video data acquisition and analysis function according to claim 1, characterized in that, The step of adjusting the set interval based on the degree of difference in image frames at any adjacent time points within an image frame of a set duration to obtain a correction interval includes: All image frames in the image frame of a set duration are converted to grayscale to obtain grayscale image frames. The average value of the grayscale value difference of corresponding pixels between two adjacent grayscale image frames is calculated as the degree of grayscale difference between the two adjacent grayscale image frames. Select the maximum value of the grayscale difference between any two adjacent grayscale image frames in all grayscale image frames, and construct a correction interval that is inversely proportional to the maximum value of the grayscale difference and is not less than the lower limit of the preset correction interval and not greater than the upper limit of the preset correction interval.

3. The edge management device with video data acquisition and analysis function according to claim 1, characterized in that, The step of determining the connected components within each image frame of the image set to be analyzed based on the pixel color similarity of adjacent pixels within the image frame includes: Calculate the absolute value of the difference between each color channel value between any two adjacent pixels in the current image frame, construct a pixel color similarity that is inversely proportional to the sum of the absolute values ​​of the differences between all color channel values, and classify two adjacent pixels whose pixel color similarity is greater than a preset pixel color similarity threshold into the same connected component.

4. The edge management device with video data acquisition and analysis function according to claim 1, characterized in that, Determining the connected component color similarity between any two adjacent image frames in the image set to be analyzed includes: Calculate the average channel value of each pixel in the current connected component under the current color channel and use it as the color feature value of the current connected component under the current color channel; In any two adjacent image frames, select one connected component and use the two selected connected components as the previous and next contrast connected components, respectively. Calculate the absolute value of the difference between the color feature values ​​of the previous and next contrast connected components in each color channel, and construct the connected component color similarity that is inversely proportional to the sum of the absolute values ​​of the differences of all color feature values.

5. The edge management device with video data acquisition and analysis function according to claim 1, characterized in that, Determining the similarity of connected component shapes between any two adjacent image frames in the image set to be analyzed includes: In any two adjacent image frames, select one connected component respectively, take the connected component with fewer edge pixels as the reference connected component, and take the other connected component as the comparison connected component. Starting from a preset edge pixel on the reference connected domain, calculate the orientation of the next edge pixel relative to the previous edge pixel in the set direction, and use the preset code corresponding to the orientation as the code of the next edge pixel, until all edge pixels on the reference connected domain are traversed to generate a reference code chain. Starting from any edge pixel in the connected component, the orientation of the next edge pixel relative to the previous edge pixel is calculated in sequence according to the set direction, and the preset code corresponding to the orientation is used as the code of the next edge pixel. This process continues until the number of encoded edge pixels reaches the total number of edge pixels in the reference connected component, and then a matching coding chain corresponding to the aforementioned edge pixel is generated. Calculate the coding chain similarity between the reference coding chain and each aligned coding chain, and use the maximum value of the coding chain similarity as the connected component shape similarity.

6. The edge management device with video data acquisition and analysis function according to claim 5, characterized in that, The similarity of the encoding chains is: ， in, This represents the coding chain similarity between the reference coding chain calculated based on the w-th connected component in the i-th video image and the u-th connected component in the (i+1)-th video image, and the y-th alignment coding chain. This represents the number of edge pixels in the connected component with fewer edge pixels between the w-th and u-th connected components. This represents the j-th encoded value in the coding chain corresponding to the w-th connected component in the i-th video image, when calculating the coding chain similarity between the reference coding chain and the y-th comparison coding chain based on the w-th connected component in the i-th video image and the u-th connected component in the (i+1)-th video image. This represents the j-th encoded value in the coding chain corresponding to the u-th connected component in the (i+1)-th video image when calculating the coding chain similarity between the reference coding chain and the y-th comparison coding chain based on the w-th connected component in the i-th video image and the u-th connected component in the (i+1)-th video image. The parameters are used to ensure that the fraction is meaningful.

7. The edge management device with video data acquisition and analysis function according to claim 1, characterized in that, Determining the position of the same customer in each image frame based on similar connected components in the image set to be analyzed includes: Similar connected components in the image set to be analyzed are marked with the same number, while non-similar connected components are marked with different numbers. The numbers corresponding to any two adjacent connected components in each image frame are used to form a number set. For any kind of similar connected component among all kinds of similar connected components, the number set with the highest frequency among all number sets containing the number corresponding to the number of ... The recombined number sets containing duplicate numbers from all similar connected component sets are merged into a fused number set. The recombined number sets containing numbers not duplicated from other recombined number sets are directly used as the fused number set. The connected components corresponding to numbers in any fused number set in the current image frame are combined to form a connected component group for that fused number set in the current image frame. The position of the connected component group of any fused number set in all image frames is used as the position of the same object in each image frame. Human-shaped objects are identified from the objects whose positions in each image frame are determined, thus completing the determination of the same customer's position in each image frame.

Images