User traffic determination method and device, server, storage medium and product

Abstract

The embodiment of the invention discloses a user traffic determination method and device, a server, a storage medium and a product. The method comprises the following steps: acquiring a to-be-used image acquired by any binocular camera in a target area; performing depth estimation on the to-be-used image based on the depth estimation model to obtain a parallax image corresponding to the to-be-used image; and processing the to-be-used image and the parallax image based on the target detection model, and outputting user traffic in the target area. According to the scheme provided by the embodiment of the invention, the depth parallax image corresponding to the target area can be determined, and then the user traffic of the target area is determined according to the parallax image and the to-be-used image, so that the user traffic determination accuracy is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of data processing technology, and in particular, to a method, device, server, storage medium and product for determining user traffic. Background Art

[0002] With the continuous development of digital transformation, the service level of various industries is also constantly improving. For some business scenarios, there are still problems such as uneven distribution of passenger flow at some service outlets over time, and long waiting time during peak hours, which affects customer experience. Therefore, determining the passenger flow information of certain service outlets is an urgent problem to be solved.

[0003] At present, the measures adopted to solve the above problems are mainly to obtain the historical user flow data of a service point, and then determine the estimated user flow of the service point at the predicted time based on the historical user flow data, so that corresponding staff or maintenance personnel can be assigned according to the estimated user flow.

[0004] The existing problems of the above method are mainly that when only relying on historical user traffic data to determine the estimated user traffic at the prediction time, there is a single factor to consider, and the impact of the real scene of the service outlet on the traffic data is not considered, resulting in inaccurate user traffic data prediction. Furthermore, when allocating staff to service outlets based on the above predicted user traffic data, there is an uneven distribution of human resources, resulting in low business processing efficiency and poor user experience. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, server, storage medium and product for determining user flow, so as to achieve the effect of improving the accuracy of determining user flow of service outlets.

[0006] In a first aspect, an embodiment of the present invention provides a method for determining user traffic, the method comprising:

[0007] Obtain an image to be used captured by any binocular camera in the target area;

[0008] Performing depth estimation on the image to be used based on a depth estimation model to obtain a disparity image corresponding to the image to be used;

[0009] The image to be used and the disparity image are processed based on a target detection model, and the user flow in the target area is output.

[0010] Furthermore, the method further comprises:

[0011] Acquire a plurality of first sample data, wherein the first sample data includes an actual left-eye image and an actual right-eye image acquired by a binocular camera;

[0012] For the first sample data, inputting the image to be input of the first sample data into the depth estimation model to be trained to obtain a left disparity image and a right disparity image; wherein the image to be input is an actual left eye image or an actual right eye image;

[0013] Determine a loss value based on the left disparity image, the right disparity image, the input image, and the first sample data, so as to update a model parameter of the depth estimation model to be trained based on the loss value;

[0014] The convergence of the loss function in the depth estimation model to be trained is taken as a training goal to obtain the depth estimation model.

[0015] Furthermore, the determining the loss value based on the left disparity image, the right disparity image, the input image and the sample data includes:

[0016] Reconstructing a predicted right-eye image based on the left disparity image and the actual right-eye image in the sample data, and reconstructing a predicted left-eye image based on the right disparity image and the actual left-eye image in the sample data;

[0017] Determining an image loss value based on the predicted left-eye image, the actual left-eye image, the predicted right-eye image, and the actual right-eye image;

[0018] Down-sampling the image to be input based on the second-order processing module in the depth estimation model to be trained to obtain a sampled image;

[0019] Determining a second-order loss value based on the sampled image, the left-view disparity image, and the right-view disparity image;

[0020] Based on the image loss value and the second-order loss value, a loss value of the sample data is determined.

[0021] Furthermore, the method further comprises:

[0022] Acquire a plurality of second sample data, wherein the second sample data includes an actual left-eye image or an actual right-eye image captured by a binocular camera;

[0023] Inputting the second sample data, and the left-eye disparity image and the right-eye disparity image corresponding to the second sample data into a detection model to be trained, so that the detection model to be trained outputs a prediction result value of the second sample data;

[0024] Determine a prediction loss value based on the prediction result value and the actual result value of the second sample data, so as to modify the model parameters in the to-be-trained detection model based on the prediction loss value;

[0025] The convergence of the loss function in the detection model to be trained is taken as the training goal to obtain the target detection model.

[0026] Furthermore, the step of inputting the second sample data, the left-eye disparity image and the right-eye disparity image corresponding to the second sample data into the detection model to be trained, so that the detection model to be trained outputs a prediction result value of the second sample data, includes:

[0027] Based on the convolutional layer in the detection model to be trained, feature extraction is performed on the second sample data, the left-eye disparity image, and the right-eye disparity image to obtain a plurality of feature images;

[0028] Determine the target features based on the feature image corresponding to each convolutional layer;

[0029] The target feature is input into the detection model to be trained to output the predicted result value of the second sample data.

[0030] Furthermore, the disparity image includes a left-eye disparity image and a right-eye disparity image, and the processing of the image to be used and the disparity image based on the target detection model to output the user flow in the target area includes:

[0031] Inputting the image to be used, the left-view disparity image, and the right-view disparity image into the target detection model, and outputting prediction information of the target area;

[0032] The prediction information is used as the user traffic in the target area.

[0033] In a second aspect, an embodiment of the present invention further provides a user flow determination device, the device comprising:

[0034] An image acquisition module is used to acquire an image to be used captured by any binocular camera in the target area;

[0035] A disparity image determination module, configured to perform depth estimation on the image to be used based on a depth estimation model to obtain a disparity image corresponding to the image to be used;

[0036] The user flow determination module is used to process the image to be used and the disparity image based on the target detection model, and output the user flow in the target area.

[0037] In a third aspect, an embodiment of the present invention further provides a server, the server comprising:

[0038] one or more processors;

[0039] A memory for storing one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the user traffic determination method as described in any one of the embodiments of the present invention.

[0041] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, a method for determining user traffic as described in any one of the embodiments of the present invention is implemented.

[0042] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, the computer program implements the user traffic determination method as described in any one of the embodiments of the present invention.

[0043] The technical solution provided by the embodiment of the present invention can obtain an image to be used captured by any camera in the target area, and then estimate the depth of the image to be used based on the depth estimation model to obtain the left-eye disparity image and the right-eye disparity image corresponding to the image to be used. Furthermore, the target detection model can process the image to be used and the disparity image to obtain the user flow in the target area, which solves the problem that the prior art only processes the image without considering the specific application scenario, resulting in the inability to effectively and accurately determine the user flow, and realizes that when the user flow is determined in combination with the depth information, the accuracy of the user flow can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required to describe the embodiments. Obviously, the drawings introduced are only drawings of a part of the embodiments to be described in the present invention, rather than all the drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A flowchart of a method for determining user traffic provided by an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a process for training a depth estimation model provided by an embodiment of the present invention;

[0047] Figure 3 A schematic diagram of a process for determining a target detection model provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of the overall structure of a method for executing user traffic prediction provided by an embodiment of the present invention;

[0049] Figure 5 A schematic diagram of a local structure corresponding to a depth estimation model provided by an embodiment of the present invention;

[0050] Figure 6 A schematic diagram of a local structure corresponding to a target detection model provided by an embodiment of the present invention;

[0051] Figure 7 A schematic diagram of the structure of a user flow determination device provided by an embodiment of the present invention;

[0052] Figure 8 A schematic diagram of the structure of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0054] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0055] Before introducing the technical solution provided by the embodiment of the present invention, an exemplary application scenario may be first described. The solution provided by the embodiment of the present invention may be applied to any situation where user flow data in a semi-enclosed area or an open area needs to be determined. The semi-enclosed area may be an indoor area with a corresponding exit, and the open area may be an outdoor area.

[0056] The solution provided in the embodiment of the present invention can be used to determine the user flow data in a certain area. Optionally, the certain area can be an area of ​​a bank branch or an express delivery branch. It should be noted that the user flow data can be understood as the number of users in a certain area.

[0057] Currently, when processing historical user traffic or images based on the network, there is a lack of understanding of the real scene, which leads to inaccurate estimation of user traffic due to occlusion between users.

[0058] Figure 1A flowchart of a method for determining user traffic provided in an embodiment of the present invention. This embodiment can be applied to any scenario where it is necessary to determine user traffic data in a certain area. The method for determining user traffic provided in this embodiment can be executed by a client, or by a server, or by a client and a server in cooperation.

[0059] like Figure 1 As shown, the method specifically comprises the following steps:

[0060] S110: Acquire an image to be used captured by any camera in the target area.

[0061] The target area may be any area whose user traffic data needs to be obtained. Optionally, if it is applied to a bank service outlet, the target area may be any bank service outlet; if it is applied to a courier station, the target area may be a specific courier station; if it is applied to a restaurant, the target area may be a restaurant. The specific target area may be set according to actual needs and is not limited in this embodiment.

[0062] Furthermore, in order to determine the user flow in the target area, at least one camera can be deployed in the target area. Optionally, the camera can be a monocular camera or a binocular camera. The specific camera type can be set by the user according to actual needs. Of course, in actual applications, the number of users in the target area can be determined based on the video stream captured by the camera. At this time, it mainly depends on the real-time viewing of the backstage staff, which has the problems of low efficiency and high labor costs. Based on this, the video stream captured by the camera can be frame-processed to obtain the image to be used that needs to be processed.

[0063] That is to say, the video stream captured by the camera can be processed by frame extraction, and the extracted video images are used as images to be used.

[0064] It should also be noted that, if the camera is a binocular camera, a video stream captured by one camera may be randomly selected, and any video frame in the video stream may be used as the image to be used.

[0065] In this embodiment, the timing of obtaining the images to be used collected by any camera in the target area can be a preset time point. For example, from 9:00 to 10:30 every day and from 1:00 to 3:00 in the afternoon are the peak hours for business handling, and the images to be used collected by any camera can be obtained every ten minutes; during non-peak hours, the images to be used collected by any camera can be obtained every half an hour. Of course, the specific collection time and collection frequency can be set according to actual needs.

[0066] Specifically, the video streams collected by any camera in the target area can be collected in sequence according to the preset image acquisition tasks, and the video streams can be processed by frame extraction to obtain the image to be used corresponding to the target area.

[0067] S120 . Perform depth estimation on the image to be used based on a depth estimation model to obtain a disparity image corresponding to the image to be used.

[0068] The depth estimation model is a pre-trained model used to perform depth estimation on the image to be used and determine the disparity image. The disparity image can be understood as the difference image between the left eye image and the right eye image after simulating left and right eyes.

[0069] Next, let's briefly explain depth estimation. Depth refers to the distance from the target object to the center of the observer's visual imaging. The observer can be a camera, and the target object mainly refers to the object or user in the target area. The purpose of depth estimation is mainly to obtain depth information from the image. Depth estimation is a basic and key technology in the field of computer vision, and is an upstream task for target detection, posture recognition, unmanned driving, and autonomous driving.

[0070] Next, let's briefly explain parallax. Figure 2 , O L and O R are the focus of the left and right cameras (the left and right cameras in the binocular camera), P is the observed target (target object), and P L ' and P R ' are the projections of point P on the left and right camera imaging planes, l is the horizontal width of the left and right images, then d = x L -x R is the parallax of point P, and the real depth of point P is z. In addition, b is the distance between the center points of the left and right cameras, called the baseline, and f is the focal length of the camera. Using the triangle similarity principle, we can know that the relationship between the parallax d and the depth z is z = (b·f) / d, and the depth z is inversely proportional to the parallax d.

[0071] In the case of understanding the above-mentioned depth estimation and disparity, a depth estimation model may be used to process an input image to obtain a disparity image of the image to be used.

[0072] Specifically, after the image to be used is acquired, the image to be used may be input into a depth estimation model, and the depth estimation model may perform depth estimation processing on the image to be used to obtain a disparity image corresponding to the image to be used.

[0073] In this embodiment, the parallax image includes a left parallax image and a right parallax image.

[0074] S130: Process the image to be used and the disparity image based on a target detection model, and output the user flow in the target area.

[0075] The target detection model is a model obtained by training to process the target image and the disparity image to predict the user flow. The user flow can be understood as the number of users in the target area.

[0076] Specifically, the image to be used and the disparity image can be input into the target detection model, and the multiple feature layers in the target detection model can process them in sequence, and finally output the user flow in the target area, thereby achieving the effect of improving the accuracy of the user flow. In this embodiment, the disparity image includes a left-eye disparity image and a right-eye disparity image, and the processing of the image to be used and the disparity image based on the target detection model to output the user flow in the target area includes: inputting the image to be used, the left-eye disparity image, and the right-eye disparity image into the target detection model, outputting the prediction information of the target area; and using the prediction information as the user flow in the target area.

[0077] The technical solution provided by the embodiment of the present invention can obtain an image to be used captured by any camera in the target area. The image to be used is input into the depth estimation model for depth estimation, and the disparity image corresponding to the image to be used can be obtained. Next, the disparity image and the image to be used are processed together based on the target detection model to obtain the estimated user flow corresponding to the target area, which solves the problem that when the user flow is determined only in combination with historical user flow or image in the prior art, the actual scene and the occlusion between users are not taken into account, resulting in inaccurate estimated user flow, and thus the corresponding human resources cannot be effectively allocated for it, resulting in low business processing efficiency. The problem is that when the user flow is comprehensively determined by combining the disparity image and the image to be used, the occlusion between users is taken into account, which can improve the effect of accurately determining the user flow, and then when human resources are allocated to it based on the accurate user flow, the effect of improving the business processing efficiency can be achieved.

[0078] Figure 2 A flow chart of a training depth estimation model provided in an embodiment of the present invention is provided. Based on the aforementioned embodiment, a depth estimation model can be trained to determine a disparity image corresponding to an image to be used based on the trained depth estimation model. The specific implementation method thereof can refer to the detailed description of the present embodiment, wherein the technical terms that are the same as or corresponding to the aforementioned embodiment are not repeated in the present embodiment.

[0079] like Figure 2 As shown, the method includes:

[0080] S210: Acquire a plurality of first sample data, wherein the first sample data includes an actual left-eye image and an actual right-eye image captured by a binocular camera.

[0081] In order to improve the accuracy of the output result of the depth estimation model, as much sample data as possible can be obtained. The sample data involved in training the depth estimation model is used as the first sample data. The first sample data may include a left eye image and a right eye image collected by a binocular camera.

[0082] It can be understood that the images collected by the binocular camera at the same time can be obtained from the open source database. Alternatively, binocular cameras are deployed in certain areas, and the left eye video stream and the right eye video stream can be collected by the binocular cameras respectively. For the left eye video stream and the right eye video stream, the video frames at the same time can be extracted and processed to obtain the left eye image and the right eye image.

[0083] The acquired left-eye image is used as the actual left-eye image, and the acquired right-eye image is used as the actual right-eye image. That is, the first sample data may include the actual left-eye image and the actual right-eye image acquired by the binocular camera at the same time.

[0084] It should be noted that the left-eye image and the right-eye image actually collected can be obtained from an open source dataset, in which case the left-eye image and the right-eye image are images corresponding to the same area. Multiple first sample data are obtained based on the above method. Next, the first sample data is divided into a training set and a test set according to a reasonable ratio. Optionally, the ratio of the training set to the test set is 8:2, and its specific ratio information can be set according to actual needs. In order to further improve the richness of the first sample data, image data can be collected by a binocular camera deployed in the target area, and the image data is mainly the actual left-eye image and the actual right-eye image. The collected image data is used as a supplementary data set to participate in the training of the depth estimation model to be trained, or to fine-tune the model parameters of the depth estimation model.

[0085] S220 . For the first sample data, input the to-be-input image of the first sample data into the to-be-trained depth estimation model to obtain a left disparity image and a right disparity image.

[0086] The image to be input is an actual left-eye image or an actual right-eye image. The model parameters in the depth estimation model to be trained are default parameters, and the depth estimation model to be trained can be trained based on the first sample data.

[0087] The sample data participating in the training of the depth estimation model to be trained is used as the first sample data. A common depth estimation method is to convert the depth estimation task into an image reconstruction process. The depth estimation model can be trained using the left eye image and the right eye image in the first sample data.

[0088] The input of the depth estimation model to be trained may be an actual left eye image or an actual right eye image, that is, the image to be input is an actual left eye image or an actual right eye image. After the image to be input is input into the depth estimation model to be trained, the convolutional neural network in the depth estimation model to be trained may process the input image to obtain a left disparity image and a right disparity image.

[0089] S230, determining a loss value based on the left disparity image, the right disparity image, the image to be input, and the first sample data, so as to update model parameters of the depth estimation model to be trained based on the loss value.

[0090] It can be understood that image reconstruction can be performed based on the left disparity image, the right disparity image, the image to be input, and the image different from the image to be input in the first sample data to obtain a reconstructed left eye image and a reconstructed right eye image. A loss value can be determined based on the reconstructed left eye image, the reconstructed right eye image, and the actual left eye image and the actual right eye image in the first sample data to update the model parameters in the depth estimation model to be trained based on the loss value.

[0091] In this embodiment, the loss value is determined based on the left disparity image, the right disparity image, the input image and the sample data, including: reconstructing a predicted right-eye image based on the left disparity image and the actual right-eye image in the sample data, and reconstructing a predicted left-eye image based on the right disparity image and the actual left-eye image in the sample data; determining an image loss value based on the predicted left-eye image, the actual left-eye image, the predicted right-eye image and the actual right-eye image; downsampling the input image to be trained based on the second-order processing module in the depth estimation model to be trained to obtain a sampled image; determining a second-order loss value based on the sampled image, the left-eye disparity image and the right-eye disparity image; determining the loss value of the sample data based on the image loss value and the second-order loss value.

[0092] It can be understood that the left disparity image and the actual right eye image in the first sample data can be reconstructed to reconstruct the right eye image, and the reconstructed right eye image can be used as the predicted right eye image. At the same time, the right disparity image and the actual left eye image in the first sample data can be reconstructed to reconstruct the left eye image, and the reconstructed left eye image can be used as the predicted left eye image. By performing loss processing on the predicted left eye image and the actual left eye image, the left eye loss value is determined, and the right eye loss value is obtained based on the loss processing of the predicted right eye image and the actual right eye image. Based on the left eye loss value and the right eye loss value, the image loss value can be determined. At this time, the image loss value mainly includes the loss value obtained after the loss processing based on the predicted left eye image, the predicted right eye image, the actual left eye image and the actual right eye image. Furthermore, it is also necessary to determine the disparity image loss value, that is, the second-order loss value.

[0093] Further, the input image can be downsampled based on the binary neural network extraction module in the depth estimation model to be trained to obtain a sampled image. Based on the sampled image, the left-eye disparity image, and the right-eye disparity image, a binary loss value can be determined. Based on the binary phase loss value and the image loss value, a loss value of the first sample data can be determined.

[0094] S240: Taking the convergence of the loss function in the depth estimation model to be trained as a training goal to obtain a depth estimation model.

[0095] It can be understood as: taking the convergence of the loss function as the training goal, the first sample data is sequentially input into the depth estimation model to be trained to update the model parameters in the depth estimation model to be trained. The depth estimation model to be trained obtained when the loss function converges is used as the depth estimation model that can be finally used.

[0096] The technical solution provided by the embodiment of the present invention can adopt the above-mentioned method to train the depth estimation model, and then when the image to be used is processed according to the depth estimation model, the disparity information corresponding to the image to be used can be obtained, so as to determine the user flow of the target area corresponding to the image to be used based on the disparity information, and the effect of determining the accuracy of the user flow can be improved.

[0097] Figure 3 A flow chart of determining a target detection model provided in an embodiment of the present invention. Based on the aforementioned embodiment, the target detection model can be first trained to process the collected images based on the target detection model to obtain the user traffic corresponding to the target area. For the specific implementation method, please refer to the detailed description of this embodiment, wherein the technical terms that are the same as or corresponding to the aforementioned embodiment will not be repeated in this implementation.

[0098] like Figure 3 As shown, the method includes:

[0099] S310: Acquire a plurality of second sample data, wherein the second sample data includes an actual left-eye image or an actual right-eye image captured by a binocular camera.

[0100] The samples used in training the target detection model may be used as the second sample data. The second sample data includes an actual left-eye image or an actual right-eye image captured by a binocular camera.

[0101] Specifically, in order to improve the accuracy of the target detection model, as many second training samples as possible may be obtained. Optionally, the second sample data may be obtained from images captured by a binocular camera deployed in a certain area.

[0102] S320: Input the second sample data, and the left-eye disparity image and the right-eye disparity image corresponding to the second sample data into the detection model to be trained, so that the detection model to be trained outputs a prediction result value of the second sample data.

[0103] It should be noted that after obtaining the second sample data, the actual left eye image or the actual right eye image in the second sample data can be input into the trained depth estimation model to obtain the left eye disparity image and the right eye disparity image. The sample content in the second sample data can be updated based on the left eye disparity image and the right eye disparity image.

[0104] The prediction result value is the predicted user traffic. The detection model to be trained is a neural network with default model parameters. The neural network can be a model corresponding to an existing detection algorithm. However, the model can integrate the depth information predicted by the depth estimation model to obtain more accurate target detection results.

[0105] After the actual left eye image or the actual right eye image, and the left disparity image and the right disparity image in the second sample data are input into the detection model to be trained, the detection model to be trained can output the prediction result value corresponding to the second sample data.

[0106] S330. Determine a prediction loss value based on the prediction result value and the actual result value of the second sample data, so as to correct the model parameters in the detection model to be trained based on the prediction loss value.

[0107] Specifically, after obtaining the predicted result value corresponding to the second sample data and the actual result value of the second sample data, the predicted loss value can be determined based on the actual result value and the predicted result value. The model parameters in the to-be-trained detection model can be corrected based on the predicted loss value.

[0108] In this embodiment, the convolutional layers in the detection model to be trained are used to extract features of the second sample data, the left-eye disparity image, and the right-eye disparity image to obtain multiple feature images; based on the feature image corresponding to each convolutional layer, target features are determined; and the target features are input into the detection model to be trained to output the predicted result value of the second sample data.

[0109] It should be noted that the processing method for each second sample data is the same. Here, the processing of one of the second sample data is taken as an example for explanation. For the remaining second sample data, please refer to the detailed explanation of this step.

[0110] The detection model to be trained includes multiple convolutional layers, each of which is used to extract features of different dimensions in the second sample data, that is, each convolutional layer can output a corresponding feature map. The feature image corresponding to each convolutional layer is used as the target feature.

[0111] Specifically, after the image in the second sample data, the left-eye disparity image and the right-eye disparity image corresponding to the second sample data are input into the detection model to be trained, each convolution layer in the detection model to be trained can process the second sample data, and each convolution layer can also extract the features corresponding to the left-eye disparity image and the right-eye disparity image. Based on the feature images corresponding to different convolution layers, the target features can be determined. Based on the processing of the target features by other neural network layers in the detection model to be trained, the prediction result value corresponding to the second sample data can be obtained.

[0112] S340: Taking the convergence of the loss function in the detection model to be trained as a training goal to obtain the target detection model.

[0113] Specifically, the convergence of the loss function in the detection model to be trained can be used as a training goal to obtain the target detection model.

[0114] The technical solution provided by the embodiment of the present invention can obtain multiple second sample data. The second sample data at this time is based on the actual left-eye image or the actual right-eye image captured by the binocular camera. At the same time, the disparity image corresponding to the second sample data can be obtained. The disparity image at this time includes the left-eye disparity image and the right-eye disparity image. Next, the second sample data and the disparity image corresponding to the second sample data are input into the detection model to be trained, so that the detection model to be trained outputs the predicted result value corresponding to the second sample data. Based on the predicted result value and the actual result value corresponding to the second sample data, a predicted loss value is rated, and the model parameters in the detection model to be trained are corrected based on the predicted loss value to obtain a target detection model, so that the target detection model used is determined by integrating the disparity image. Furthermore, when the captured image is input into the target detection model, the model can recognize the depth information in the image, thereby improving the accuracy of user force prediction.

[0115] Figure 4 The overall structure diagram of the method for executing user flow prediction provided by an embodiment of the present invention is as follows: Figure 5 A schematic diagram of a local structure corresponding to a depth estimation model provided in an embodiment of the present invention, Figure 6 The local structural diagram corresponding to the target detection model provided in the embodiment of the present invention is explained based on the above-mentioned embodiment by taking the specific application scenario of determining the user flow corresponding to the bank branch as an example, wherein the technical terms that are the same as or corresponding to the above-mentioned embodiment are not repeated in this embodiment.

[0116] Next, combine Figure 4 , Figure 5 as well as Figure 6 To explain the solution provided by the embodiment of the present invention:

[0117] The data processing device is responsible for processing the data required by the depth estimation model to be trained and the detection model to be trained to obtain a plurality of first sample data and a plurality of second sample data.

[0118] The data processing device can be determined based on an existing data set, and optionally, obtains the actual left-eye image and the actual right-eye image captured by multiple binocular cameras. After the acquisition is completed, it can be divided into a training set and a test set in proportion, and in practice, a ratio of 8:2 can be used for division. By collecting image data with binocular cameras deployed at bank outlets, the collected image data can be used as a supplementary data set to fine-tune the trained model, so that the obtained model is more suitable for user flow prediction in bank outlet scenarios.

[0119] The core of the depth estimation model (depth estimation device) is image reconstruction, such as Figure 3As shown, the depth estimation model to be trained can be trained using the first sample data. The actual left-eye image or the actual right-eye image in the first sample data can be input into the depth estimation model to be trained. The convolutional neural network in the depth estimation model to be trained can process the input image and output a left disparity image and a right disparity image. The left disparity image can be fused with the actual right-eye image to reconstruct a reconstructed left-eye image; correspondingly, the right disparity image can be fused with the actual left-eye image to reconstruct a reconstructed right-eye image. Since the model parameters in the depth estimation model to be trained have not been trained, there are certain differences between the reconstructed left-eye image and the reconstructed right-eye image and the actually captured images.

[0120] It can be understood that the depth estimation algorithm used in the existing depth estimation model may lack constraints on the relationship between the depths of different pixels in the image, resulting in the generated depth image being not smooth enough. Considering that the image depth is continuously distributed, and the conditional random field can establish a local and global relationship, it has good applicability to continuous distribution. Therefore, the solution provided by the embodiment of the present invention can combine the conditional random field with the convolutional neural network to model and constrain the intrinsic relationship between each pixel in the image, so as to achieve the purpose of main inner line and smoothness at the same time.

[0121] Here, the depth estimation model to be trained includes a unary neural network feature extraction module, a binary neural network feature extraction module and a feature fusion module. The unary neural network feature extraction module is used to extract unary features, that is, mainly to perform loss processing on the predicted left-eye image, the actual left-eye image, the predicted right-eye image and the actual right-eye image to obtain an image loss value. At the same time, the binary neural network extraction module is used to downsample the input image to obtain a sampled image. Based on the sampled image, the left-eye disparity image and the right-eye disparity image, a binary loss value is determined. Based on the image loss value and the binary loss value, the loss value corresponding to the first sample data can be determined to correct the model parameters in the depth estimation model to be trained based on the loss value. The depth estimation model to be trained obtained when the loss function converges is used as a usable depth estimation model.

[0122] The target detection model (target detection device) can be found in Figure 6 The target detection model can adopt the existing model algorithm, but the target detection model can integrate the depth information predicted by the depth estimation model to obtain more accurate target detection results.

[0123] The target detection model can use a convolutional neural network to extract features from the image, and use pyramid pooling to obtain the input second sample data, the left disparity image corresponding to the second sample data, and the right disparity image for processing to obtain feature images of different scales. Then, the feature information of different scales is fused with the depth information output by the depth estimation device (i.e., the depth features extracted from different dimensions), and up-sampled using a convolutional neural network. Regularization and activation functions are usually used to process the fused target features to obtain predicted user traffic. According to the predicted user traffic and the actual user traffic corresponding to the second sample data, a loss value is determined to correct the model parameters in the detection model based on the loss value. The convergence of the loss function in the detection model is used as the training goal to obtain a target detection model.

[0124] After obtaining the target detection model, the predicted user flow in the bank branch can be determined based on the target detection model. That is, on the basis of obtaining accurate customer flow data, through processing through data analysis algorithms such as genetic algorithms, the human resource allocation of the branch can be optimized and the service level of the branch can be improved.

[0125] The technical solution provided by the embodiment of the present invention, by introducing depth information, enables the target detection model to better overcome problems such as occlusion and overlap, improves detection accuracy, and better fits the task of detecting customer flow at bank branches. Furthermore, the technical solution provided by the embodiment of the present invention not only introduces depth information to improve the target detection effect, but also takes into account the correlation of depth information between different pixels on the image. By combining the convolutional neural network and the conditional random field, the correlation between the depths of different pixels in the image is constrained, so that the generated depth map is smoother and more accurate than the existing depth estimation. Correspondingly, the predicted user flow is more accurate and reliable.

[0126] The following is an embodiment of a user flow determination device provided by an embodiment of the present invention. The device and the user flow determination method of the above embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the user flow determination device, reference can be made to the embodiment of the above user flow determination method.

[0127] Figure 7 This is a structural diagram of a user flow determination device provided by an embodiment of the present invention. The device specifically includes: an image acquisition module 410, a parallax image determination module 420 and a user flow determination module 430.

[0128] Among them, the image acquisition module 410 is used to acquire an image to be used captured by any binocular camera in the target area; the disparity image determination module 420 is used to perform depth estimation on the image to be used based on the depth estimation model to obtain the disparity image corresponding to the image to be used; the user flow determination module 430 is used to process the image to be used and the disparity image based on the target detection model, and output the user flow in the target area.

[0129] The technical solution provided by the embodiment of the present invention can obtain an image to be used captured by any camera in the target area, and then estimate the depth of the image to be used based on the depth estimation model to obtain the left-eye disparity image and the right-eye disparity image corresponding to the image to be used. Furthermore, the target detection model can process the image to be used and the disparity image to obtain the user flow in the target area, which solves the problem that the prior art only processes the image without considering the specific application scenario, resulting in the inability to effectively and accurately determine the user flow, and realizes that when the user flow is determined in combination with the depth information, the accuracy of the user flow can be improved.

[0130] On the basis of the above technical solution, the device further comprises:

[0131] A first sample data acquisition module, used to acquire a plurality of first sample data, wherein the first sample data includes an actual left-eye image and an actual right-eye image acquired based on a binocular camera;

[0132] A disparity image determination module, for the first sample data, inputting the image to be input of the first sample data into the depth estimation model to be trained to obtain a left disparity image and a right disparity image; wherein the image to be input is an actual left eye image or an actual right eye image;

[0133] A first parameter updating module, configured to determine a loss value based on the left disparity image, the right disparity image, the input image, and the first sample data, so as to update a model parameter of the depth estimation model to be trained based on the loss value;

[0134] The depth estimation model determination module is used to take the convergence of the loss function in the depth estimation model to be trained as a training target to obtain the depth estimation model.

[0135] Based on the above technical solutions, the first parameter updating module includes:

[0136] A predicted image determination unit, configured to reconstruct a predicted right-eye image based on the left disparity image and the actual right-eye image in the sample data, and to reconstruct a predicted left-eye image based on the right disparity image and the actual left-eye image in the sample data;

[0137] A first loss value determining unit, configured to determine an image loss value based on the predicted left-eye image, the actual left-eye image, the predicted right-eye image, and the actual right-eye image;

[0138] A sampling image determining unit, configured to perform down-sampling processing on the image to be input based on the second-order processing module in the depth estimation model to be trained to obtain a sampling image;

[0139] A second loss value determining unit, configured to determine a second-order loss value based on the sampled image, the left-view disparity image, and the right-view disparity image;

[0140] The third loss value determining unit is used to determine the loss value of the sample data based on the image loss value and the second-order loss value.

[0141] On the basis of the above technical solutions, the device further includes:

[0142] A second sample data acquisition module, used to acquire a plurality of second sample data, wherein the second sample data includes an actual left-eye image or an actual right-eye image acquired by a binocular camera;

[0143] A prediction result value determination module, used for inputting the second sample data, the left-eye disparity image and the right-eye disparity image corresponding to the second sample data into a detection model to be trained, so that the detection model to be trained outputs a prediction result value of the second sample data;

[0144] A parameter correction module, used to determine a prediction loss value based on the prediction result value and the actual result value of the second sample data, so as to correct the model parameters in the to-be-trained detection model based on the prediction loss value;

[0145] The target detection model determination module is used to take the convergence of the loss function in the detection model to be trained as the training goal to obtain the target detection model.

[0146] On the basis of the above technical solutions, the prediction result value determination module includes:

[0147] A feature image determination unit, configured to extract features from the second sample data, the left-eye disparity image, and the right-eye disparity image based on the convolutional layer in the detection model to be trained, to obtain a plurality of feature images;

[0148] A target feature determination unit, used for determining target features based on a feature image corresponding to each convolutional layer;

[0149] A prediction result value determination unit is used to input the target feature into the detection model to be trained to output the prediction result value of the second sample data.

[0150] Based on the above technical solution, the traffic prediction module is used to

[0151] Inputting the image to be used, the left-view disparity image, and the right-view disparity image into the target detection model, and outputting prediction information of the target area;

[0152] The prediction information is used as the user traffic in the target area.

[0153] The technical solution provided by the embodiment of the present invention can obtain an image to be used captured by any camera in the target area. The image to be used is input into the depth estimation model for depth estimation, and the disparity image corresponding to the image to be used can be obtained. Next, the disparity image and the image to be used are processed together based on the target detection model to obtain the estimated user flow corresponding to the target area, which solves the problem that when the user flow is determined only in combination with historical user flow or image in the prior art, the actual scene and the occlusion between users are not taken into account, resulting in inaccurate estimated user flow, and thus the corresponding human resources cannot be effectively allocated for it, resulting in low business processing efficiency. The problem is that when the user flow is comprehensively determined by combining the disparity image and the image to be used, the occlusion between users is taken into account, which can improve the effect of accurately determining the user flow, and then when human resources are allocated to it based on the accurate user flow, the effect of improving the business processing efficiency can be achieved.

[0154] The user flow determination device provided in the embodiment of the present invention can execute the user flow determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the user flow determination method.

[0155] It is worth noting that in the embodiment of the above-mentioned user flow determination device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0156] Figure 8 A schematic diagram of the structure of a server provided in an embodiment of the present invention. Figure 8 A block diagram of an exemplary server 12 suitable for use in implementing embodiments of the present invention is shown. Figure 8 The server 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0157] like Figure 8As shown, the server 12 is in the form of a general purpose computing device. The components of the server 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0158] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0159] The server 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the server 12, including volatile and non-volatile media, removable and non-removable media.

[0160] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 8 not shown, usually called a "hard drive"). Although Figure 8 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0161] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0162] The server 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the server 12, and / or may communicate with any device that enables the server 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. In addition, the server 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the server 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the server 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0163] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing the steps of the user flow determination method provided in the first embodiment of the present invention, the method comprising:

[0164] Obtain an image to be used captured by any camera in the target area;

[0165] Performing depth estimation on the image to be used based on a depth estimation model to obtain a disparity image corresponding to the image to be used;

[0166] The image to be used and the disparity image are processed based on a target detection model, and the user flow in the target area is output.

[0167] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the user flow determination method provided by any embodiment of the present invention.

[0168] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of the method for determining user traffic provided in the above embodiment of the present invention are implemented. The method includes:

[0169] Obtain an image to be used captured by any camera in the target area;

[0170] Performing depth estimation on the image to be used based on a depth estimation model to obtain a disparity image corresponding to the image to be used;

[0171] The image to be used and the disparity image are processed based on a target detection model, and the user flow in the target area is output.

[0172] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0173] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0174] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0175] Computer program code for performing the operation of embodiments of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0176] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for determining user traffic, characterized in that: include: Obtain an image to be used captured by any camera in the target area; Performing depth estimation on the image to be used based on a depth estimation model to obtain a disparity image corresponding to the image to be used; The image to be used and the disparity image are processed based on a target detection model, and the user flow in the target area is output.

2. The method according to claim 1, characterized in that The method further comprises: Acquire a plurality of first sample data, wherein the first sample data includes an actual left-eye image and an actual right-eye image acquired by a binocular camera; For the first sample data, inputting the image to be input of the first sample data into the depth estimation model to be trained to obtain a left disparity image and a right disparity image; wherein the image to be input is an actual left eye image or an actual right eye image; Determine a loss value based on the left disparity image, the right disparity image, the image to be input, and the first sample data, so as to update a model parameter of the depth estimation model to be trained based on the loss value; The convergence of the loss function in the depth estimation model to be trained is taken as a training goal to obtain the depth estimation model.

3. The method according to claim 2, characterized in that The determining of the loss value based on the left disparity image, the right disparity image, the input image, and the first sample data includes: Reconstructing a predicted right-eye image based on the left disparity image and the actual right-eye image in the first sample data, and reconstructing a predicted left-eye image based on the right disparity image and the actual left-eye image in the first sample data; Determining an image loss value based on the predicted left-eye image, the actual left-eye image, the predicted right-eye image, and the actual right-eye image; Down-sampling the image to be input based on the binary neural network extraction module in the depth estimation model to be trained to obtain a sampled image; Determining a binary term loss value based on the sampled image, the left-view disparity image, and the right-view disparity image; Based on the image loss value and the binary term loss value, a loss value of the first sample data is determined.

4. The method according to claim 1, characterized in that: The method further comprises: Acquire a plurality of second sample data, wherein the second sample data includes an actual left-eye image or an actual right-eye image captured by a binocular camera; Inputting the second sample data, and the left-eye disparity image and the right-eye disparity image corresponding to the second sample data into a detection model to be trained, so that the detection model to be trained outputs a prediction result value of the second sample data; Determine a prediction loss value based on the prediction result value and the actual result value of the second sample data, so as to modify the model parameters in the to-be-trained detection model based on the prediction loss value; The convergence of the loss function in the detection model to be trained is taken as the training goal to obtain the target detection model.

5. The method according to claim 4, characterized in that The step of inputting the second sample data, the left-eye disparity image and the right-eye disparity image corresponding to the second sample data into the detection model to be trained, so that the detection model to be trained outputs a prediction result value of the second sample data, comprises: Based on the convolutional layer in the detection model to be trained, feature extraction is performed on the second sample data, the left-eye disparity image, and the right-eye disparity image to obtain a plurality of feature images; Determine the target features based on the feature image corresponding to each convolutional layer; Based on the target feature, a prediction result value of the second sample data is determined.

6. The method according to claim 1, characterized in that The disparity image includes a left-eye disparity image and a right-eye disparity image, and the processing of the to-be-used image and the disparity image based on the target detection model to output the user flow in the target area includes: Inputting the image to be used, the left-view disparity image, and the right-view disparity image into the target detection model, and outputting prediction information of the target area; The prediction information is used as the user traffic in the target area.

7. A user flow determination device, characterized in that: include: An image acquisition module is used to acquire an image to be used captured by any binocular camera in the target area; A disparity image determination module, configured to perform depth estimation on the image to be used based on a depth estimation model to obtain a disparity image corresponding to the image to be used; The user flow determination module is used to process the image to be used and the disparity image based on the target detection model, and output the user flow in the target area.

8. A server, characterized in that: The server includes: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the user traffic determination method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for determining user traffic as described in any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the user flow determination method as described in any one of claims 1 to 7.

Images