Image processing method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application provide an image processing method and device, electronic equipment and computer readable storage medium. The method comprises: determining position information of at least one target object in a target region according to a plurality of images of the target region; generating trajectory information of the at least one target object according to the position information; fitting the trajectory information of each target object to generate a fitting curve; and determining an action event of the target object according to the fitting curve. Embodiments of the present application can utilize the fitting curve based on the trajectory information to determine the action event of the target object, such as the customer, thereby eliminating the need for manual calibration of auxiliary lines based on trajectory information in the prior art, greatly improving efficiency and accuracy of determination.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method and device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] With the development of image processing technology, various applications based on image processing technology have been gradually used in various aspects of people's life. In particular, in the field of data statistics based on image recognition, statistical data can be obtained by recognizing target objects in images, thereby assisting users in making various business decisions. For example, when a mall operator wants to count the number of customers in the mall, the mall operator can determine the action trajectory of the customers by means of image processing technology based on image recognition, thereby helping the mall operator to determine and adjust the business strategy.

[0003] In such a data statistics process based on target recognition, after obtaining the information of the action trajectory of the target object, it is usually necessary to further draw an auxiliary line for event determination to generate corresponding event information when the action trajectory of the target object crosses the auxiliary line, such as the event information of entering or exiting the mall or the floor of the mall.

[0004] However, in the prior art, after the image recognition obtains the information of the action trajectory of the target object, the auxiliary line is usually manually drawn and maintained by manual intervention. This not only consumes a lot of manpower, but also because the subjective factors of people are introduced, the drawing accuracy of the auxiliary line is also affected by the subjectivity, and the efficiency is very low, which is difficult to meet the demand for a large number of customer flow statistics. SUMMARY

[0005] The embodiments of the present application provide an image processing method and device, an electronic device, and a computer readable storage medium to solve the problem of manual calibration of auxiliary lines and low efficiency in the prior art.

[0006] To achieve the above-mentioned purpose, the embodiments of the present application provide an image processing method, comprising:

[0007] determining position information of at least one target object in a target region according to a plurality of images of the target region;

[0008] generating trajectory information of the at least one target object according to the position information;

[0009] performing fitting processing on the trajectory information of each target object to generate a fitting curve;

[0010] determining an action event of the target object according to the fitting curve.

[0011] The embodiments of the present application also provide an image processing device, comprising:

[0012] determining position information of at least one target object in the target area according to a plurality of images of the target area;

[0013] generating trajectory information of the at least one target object according to the position information;

[0014] fitting the trajectory information of each target object to generate at least one fitting curve;

[0015] determining an action event of the target object according to the fitting curve.

[0016] The embodiment of the present application further provides an electronic device, comprising:

[0017] a memory, configured to store a program;

[0018] a processor, configured to execute the program stored in the memory, and the program performs the image processing method provided by the embodiment of the present application when executed.

[0019] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program executable by a processor, and the program is executed by the processor to implement the image processing method provided by the embodiment of the present application.

[0020] The image processing method and device, the electronic device and the computer readable storage medium provided by the embodiment of the present application can determine the position information of the target object in the target area according to the identification processing of the images of the target area and further generate the trajectory information, so as to determine the action event of the target object according to the fitting curve generated by fitting the trajectory information of the target object, and therefore, the embodiment of the present application can utilize the fitting curve based on the trajectory information to perform the processing such as the determination of the action event of the target object, so as to eliminate the problem that the auxiliary line needs to be manually calibrated based on the trajectory information in the prior art, and in particular, the action event of the target object in the target area can be determined based on the trajectory information and the fitting curve, which not only greatly improves the efficiency and the accuracy of the determination, but also provides a more accurate reference for the management of the target area or the service provider providing the service for the target object, and in addition, the spline curve of the trajectory line can be obtained based on the fitting of the trajectory information, so that the spline curve is used as a constraint condition for fitting, which not only reduces the action trajectory to be processed, but also greatly reduces the calculation amount of the fitting through the constraint of the spline curve, and improves the accuracy of the fitting result.

[0021] The above description is only a summary of the technical solutions of the present application. In order to enable one skilled in the art to better understand the technical means of the present application, the present application can be implemented according to the content of the description, and in order to enable the above and other purposes, characteristics and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not meant to limit the present application. Moreover, the same reference numerals in the attached drawings indicate the same or similar components. In the drawings:

[0023] Figure 1 An application scenario diagram of the image processing method provided by the embodiments of the present application is shown in the figure.

[0024] Figure 2 A flowchart of one embodiment of the image processing method provided by the present application is shown in the figure.

[0025] Figure 3 A flowchart of another embodiment of the image processing method provided by the present application is shown in the figure.

[0026] Figure 4 A structure diagram of an embodiment of the image processing device provided by the present application is shown in the figure.

[0027] Figure 5 A structure diagram of an embodiment of the electronic device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0028] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0029] Embodiment One

[0030] The scheme provided by the embodiments of the present application can be applied to any image processing system with image processing capability. Figure 1 An application scenario diagram of the image processing method provided by the embodiments of the present application is shown in the figure. Figure 1 The scenario shown is only one of the examples of the scenarios to which the technical solutions of the present application can be applied.

[0031] With the development of computer technology, people can use computers to perform complex calculations, such as identifying target objects in images, and then based on the results of image recognition, various work can be performed. For example, in the operation of a shopping mall, the operator of the shopping mall usually needs to count the flow of customers in the shopping mall, such as when customers are more frequent, or the area where customers often visit, etc., so that the operation strategy of the shopping mall can be adjusted according to such flow conditions, such as adjusting the business hours or adjusting the types of goods in certain areas, etc. Therefore, in the prior art, computer technology has been used to process images of various areas of a shopping mall to identify the positions of target objects, i.e. customers, in the area, and then by processing a series of images within a period of time, the movement trajectory information of each customer can be obtained. For example, as shown in Figure 1 , by processing a series of images of a target area 1 in a specific time period, a series of position points of customers 1-3 in the target area 1 can be obtained, so that these position points can constitute the trajectory information of customer 1-3. Therefore, in the prior art, auxiliary lines are usually drawn by specialized personnel based on the trajectory information for event determination. For example, in the scenario shown in Figure 1 , the operator of the shopping mall wants to count the flow of customers in and out of the shopping mall, so auxiliary lines can be drawn near the entrance, so that when the trajectory information of one of the customers 1-3 crosses the auxiliary line, it can be determined that the customer enters or exits the shopping mall once. However, such manual calibration, due to the subjective factors of humans, may result in, for example, if the auxiliary line is set unreasonably, customer 2 may also leave from the exit, but his trajectory line does not cross the manually set auxiliary line, so there will be a missed judgment, or customer 3 may repeatedly cross the previously set auxiliary line due to repeated walking near the exit, so there will be repeated counting, which reduces the accuracy of the statistics. Or the results need to be checked manually by the staff periodically, and when it is found that the previously set auxiliary line leads to a missed judgment of the trajectory line of customer 2, manual adjustment is made. However, this makes the processing very inefficient.

[0032] Therefore, in the embodiments of the present application, after obtaining the trajectory information of the customers, fitting is performed based on the trajectory information such as position points or trajectory lines, so as to obtain the fitting curve of the movement of each customer, such as the fitting curve 1-3 shown in Figure 1 . By fitting the trajectory information of the customers to obtain the fitting curve, not only can a curve that more intuitively reflects the movement trajectory of the customers be obtained, such as the trajectory caused by the repeated walking of customer 3 at the exit can be excluded. And further, the present application can determine the movement events of the customers, such as the events of leaving or entering the shopping mall, according to the fitting curve that more accurately reflects the movement route of the customers, especially the starting point and the end point.

[0033] The image processing scheme provided by the embodiments of the present application determines the position information of the target object in the target region through the recognition processing of the image of the target region, and further generates trajectory information, so as to determine the action event of the target object according to the fitting curve generated by fitting the fitting curve of the target object. Therefore, the embodiments of the present application can utilize the fitting curve based on the trajectory information to perform the processing such as the determination of the action event of the target object (for example, the customer), thereby eliminating the problem that the auxiliary line needs to be calibrated manually based on the trajectory information in the prior art, and greatly improving the efficiency and the accuracy of the determination.

[0034] The above embodiments are descriptions of the technical principles and exemplary application frameworks of the embodiments of the present application. The specific technical solutions of the embodiments of the present application are further described in detail through multiple embodiments.

[0035] Embodiment Two

[0036] Figure 2 An embodiment of the image processing method provided by the present application is shown in a flowchart. The execution subject of the method can be various terminals or server devices with image processing capability, or devices or chips integrated on these devices. As shown in the figure, the image processing method includes the following steps: Figure 2

[0037] S201, determining position information of at least one target object in a target region according to a plurality of images of the target region.

[0038] In the embodiments of the present application, the position information of at least one target object in the region can be determined through image recognition from a series of images obtained for a pre-set target region. For example, in the scenario shown in Figure 1 , three customers are recognized in the target region 1, so the positions of the three customers in the images of the target region are determined in step S201. For example, Figure 1 The scenario shown in is the case of a shopping mall, and the three customers move in the shopping mall to view goods or perform shopping activities and the like within a certain time period. Therefore, the three customers can have a plurality of position points in the series of images of the region, and the position point information of each customer at different time points can be determined respectively through image recognition techniques such as person recognition on the images.

[0039] S202, generating trajectory information of at least one target object according to the position information.

[0040] ​After the location information of each customer is determined in step S201, the trajectory information of each target object, i.e. customer, can be generated according to the location information in step S202. For example, in step S202, a series of location points of the customer in images with predetermined time intervals can be selected in a high-frequency sampling manner to generate the trajectory information, e.g. trajectory line, of the customer. For example, in step S202, the time interval for selecting a series of images can be determined according to the attributes or characteristics of the target region or target object. For example, in the case that the target region is a certain floor of a shopping mall, the time interval for selecting a series of images can be set according to the characteristics of the goods sold on the floor. For example, in a children’s floor, the time interval can be set to be shorter, e.g. one image per second or ten seconds, during weekend time, and longer, e.g. ten minutes, during weekdays, etc.; or in a dining floor, the collection time interval can be set to be shorter during lunch and dinner time periods, and longer during other time periods. It is noted that since the action of a customer in a shopping mall is usually arbitrary, i.e. the customer does not move along a certain route, but can move back and forth between two or several location points as shown in FIG. 1, the trajectory line drawn by selecting the location points in images with predetermined time intervals in step S202 can be irregular or even somewhat messy. Figure 1

[0041] Therefore, in the prior art, the auxiliary lines are usually drawn manually for the trajectory lines, but as mentioned above, the location points obtained by sampling tend to result in irregular trajectory lines due to the arbitrariness of the movement of the customer, and thus when there are a large number of target objects and the location information is very complex in step S201, the manually drawn auxiliary lines often have problems such as missing judgment or repeated statistics, resulting in low accuracy of the final result, and the manual intervention for setting also results in that the whole process from image recognition to result statistics cannot be automatically processed, which is low in efficiency.

[0042] S203, fitting the trajectory information of each target object to generate a fitted curve of the target object.

[0043] ​Therefore, after the trajectory information of the target objects in the target region is obtained in step S202, the fitting curve of the target objects can be generated by fitting processing according to the trajectory information of the target objects identified in the target region. In the embodiment of the present application, since the positions of each target object, i.e. the customers, in the image can be identified in step S201, the trajectories of each customer can be obtained in step S202, for example, a relatively disordered trajectory composed of many trajectories can be obtained in step S202. Such a disordered trajectory can only be marked by an auxiliary line by manual visual and experience-based assistance in the prior art to determine the action of the customer, for example, the entering and exiting behavior. Therefore, in the embodiment of the present application, the disordered trajectories formed in step S203 can be fitted into, for example, a smaller number of fitting curves. For example, in the embodiment of the present application, the trajectories with a small distance, i.e. very close to each other, can be fitted to generate a spline curve, or the trajectories of a plurality of customers walking together can be fitted to generate a trajectory to represent the moving trajectory of the group of customers, so that the calculation amount of subsequent processing, for example, determination processing, can be greatly reduced, and the processing efficiency can be improved.

[0044] S204, determining the action event of the target object according to the fitting curve.

[0045] In the embodiment of the present application, after the fitting curve based on the trajectory of the target object in the target region is obtained in step S203, the action event of the target object can be further determined according to the fitting curve in step S204. For example, since the trajectory line in step S202 is processed into a smaller number of fitting curves by fitting processing in step S203, the entering and exiting passageway of the customer in the mall can be determined according to the trend and concentration degree of the fitting curves in step S204, and the entering and exiting behavior of the customer to the mall can be determined. In the embodiment of the present application, the action event of the corresponding target object, for example, the entering and exiting behavior, can be further determined according to the crossing event, for example, the intersection of the trajectory information and the fitting curve in step S203.

[0046] The image processing method provided by the embodiment of the present application determines the position information of the target object in the target region and further generates the trajectory information by identifying the image of the target region, and determines the action event of the target object according to the fitting curve generated by fitting the trajectory of the target object, so that the embodiment of the present application can utilize the fitting curve based on the trajectory information to determine the action event of the target object, for example, the customer, and thus eliminates the problem that the auxiliary line needs to be marked by manual based on the trajectory information in the prior art, greatly improving the efficiency and the accuracy of determination.

[0047] Embodiment Three

[0048] Figure 3The flowchart illustrates another embodiment of the image processing method provided in this application. The subject executing this method can be various terminal or server devices with image processing capabilities, or it can be a device or chip integrated into these devices. Figure 3 As shown, the image processing method includes the following steps:

[0049] S301, determine the location information of at least one target object in the target area based on multiple images of the target area.

[0050] In this embodiment of the application, image recognition can be performed on a series of images obtained for a pre-set target area to determine the location information of at least one target object in the image within that area. For example, in Figure 1 In the scenario shown, three customers are identified in target area 1, therefore, in step S201, the positions of these three customers in the image targeting that target area are determined. For example, Figure 1 In the scenario shown, where the three customers are walking around the mall, looking at products, or shopping within a certain time period, these three customers can have multiple location points in this series of images of that area. Therefore, by using image recognition technology, such as personnel recognition, the location information of the three customers at different times can be determined separately.

[0051] S302 generates the trajectory line of the target object using high-frequency sampling based on the location information.

[0052] After determining the location information of each customer in step S301, in step S302, trajectory lines can be generated for each target object, i.e., the customer, based on this location information, for example, using high-frequency sampling. For example, in step S302, a series of location points of the customer in an image with a predetermined time interval can be selected using high-frequency sampling to generate the customer's trajectory line.

[0053] In this embodiment, after obtaining the trajectory line of the target object in step S302, the generated trajectory line can be further displayed visually. For example, the generated trajectory line can be displayed to users, such as the administrator of the target area or the service provider offering services to the target object, through a monitor, and instructions given by the user regarding the displayed trajectory line can be received. In this embodiment, the display of the trajectory line can be adjusted according to the user's needs through such display and interaction with the user. For example, the instructions given by the user can include at least instructions related to the visual display of the displayed trajectory line, such as zooming in or calibrating a certain part of the trajectory line, so that subsequent visualization and processing of the trajectory line can be carried out in response to such user instructions.

[0054] S3031, fitting the trajectory information of each target object to generate a fitted curve of the target object.

[0055] Therefore, after obtaining the trajectory information of the target objects in step S302, the fitted curve can be generated by fitting the trajectory information of the target objects recognized in the target area. In the embodiment of the present application, since the position of each target object, i.e. the customer, in the image can be recognized in general, the trajectory of each customer can be obtained in step S202 accordingly, for example, a relatively disordered trajectory composed of many trajectories can be obtained in step S202. Such a disordered trajectory can only be manually marked with auxiliary lines based on vision and experience to determine the action of the customer, such as the in-out behavior, in the prior art. Therefore, in the embodiment of the present application, the disordered trajectories formed can be fitted into a smaller number of fitted curves in step S3031.

[0056] S3032, grouping the trajectory information of at least one target object by using a clustering algorithm to generate a plurality of groups of trajectory information.

[0057] In the embodiment of the present application, after obtaining the trajectory lines of the target objects in step S302, since the trajectory lines are generated based on the real action positions of the customers in the target area, the generated trajectory lines are often very disordered due to the randomness of such actions, i.e. the customer can repeatedly walk in a very small area, so that the trajectory lines in this area appear as a disordered line bundle. Therefore, in step S3032, the trend of the trajectory lines in step S302 can be used to group the trajectory lines by using an unsupervised clustering algorithm, for example, trajectory 1 and 2 are trajectory lines converging to the upper side, and trajectory 3 is a trajectory line converging to the lower side, so that trajectory 1 and 2 can be grouped to generate a trajectory line, for example, to represent a concentrated passage of the customers in the target area, and the trend of the passage can be determined by the direction of the trajectory 1 and the trajectory 2, for example, left, right or the opposite direction, for example, if the trajectory 1 and the trajectory 2 are in the same direction, the trend of the passage can be determined as a one-way direction towards the direction, and if the trajectory 1 and the trajectory 2 are in different directions, the trend of the passage can be determined as a two-way direction.

[0058] S3033, fitting the plurality of groups of trajectory information to obtain a spline curve of the trajectory line corresponding to the trajectory information.

[0059] Therefore, on the basis of generating the multiple sets of trajectory information in step S3032, the multiple sets of trajectory information can be fitted in step S3033 to obtain a spline curve of the trajectory line corresponding to the set of trajectory information. For example, in the embodiment of the present application, the grouped trajectory lines can be first smoothed, i.e., spline smoothing, so that the spline curve of the trajectory line can be obtained.

[0060] S3034, generating a fitted curve by taking the multiple sets of trajectory information as training data and taking the spline curve as a constraint condition.

[0061] In step S3034, the grouped trajectory information generated in S3032 can be taken as training data and the spline curve in step S3033 can be taken as a constraint condition for fitting processing. For example, in the embodiment of the present application, the loss function can be designed according to the category of the auxiliary line to be calibrated, i.e., a loss model is constructed, and the loss model is trained by using the grouped trajectory lines in S3032 which can reflect the action characteristics of the target region, such as the passage, to ensure that the trajectory information obtained in step S302 input into the model can achieve the maximum data hit rate. For example, when calibrating the access auxiliary line, the auxiliary line category can be first set as the access auxiliary line, and then the grouped trajectory information in step S3032 is used as training data, the spline curve in step S3033 is used as a constraint equation to fit the auxiliary line using the loss function, and the training data is used to check whether the auxiliary line fitted in the current round can satisfy that the trajectory information determined in step S302 can all be determined as access events, and if the trajectory information determined as access events is not enough, the fitting processing can be repeated until the determined auxiliary line can meet the accuracy requirement. Thus, the determined auxiliary line is finally determined as the fitted curve.

[0062] S304, determining the action event of the target object according to the trajectory information and the fitted curve.

[0063] In the embodiment of the present application, after obtaining the fitted curve of the trajectory of the target object in the target region, the action event of the target object can be further determined according to the fitted curve. Therefore, the access passage of the customer in the mall can be determined according to the trend and concentration degree of the fitted curve, and then the access behavior of the customer to the mall can be determined. In the embodiment of the present application, the action event of the corresponding target object, such as the access behavior, can be further determined according to the crossing event, such as the intersection, of the trajectory information in step S3031 and the fitted curve.

[0064] S3051, determining the starting point or the ending point of the trajectory line of the target object according to the trajectory information of the target object and the starting point or the ending point information in the position information of the target object.

[0065] S3052, obtaining a fitting region according to the fitting curve.

[0066] In the embodiment of the present application, after the plurality of trajectory information is fitted into a smaller number of fitting curves in step S3031, since such fitting curves can usually more accurately show the statistical trajectory of the customers who can walk in the region, the fitting region can be further determined according to the fitting curve in step S3051. For example, the starting point or the ending point in the trajectory line can be determined according to the starting point or the ending point information in the position information obtained in step S301 and the trajectory information in step S302, so as to construct the fitting region based on the fitting curve around or passing through the starting point or the ending point.

[0067] S3053, taking the starting point or the ending point of the target object appearing or leaving the fitting region as the action event of the target object in the target region.

[0068] After the fitting region is determined in step S3052, the starting point or the ending point of the position information of the target object can be taken as the action event of the target object, i.e., the customer entering or leaving the target region, appearing or leaving the determined fitting region in step S3053. Therefore, the entering and leaving behavior of the customer can be determined with the region as a determination range.

[0069] In addition, after the fitting region is determined, the determination of the action event can also be performed similarly to step S3053 with the fitting region as a kind of field of view boundary. For example, the boundary of the fitting region determined in step S3052 can be taken as the field of view boundary, and the starting point or the ending point of the target object entering or leaving the field of view boundary can be taken as the action event of the customer entering or leaving the region.

[0070] In the embodiment of the present application, after the action event of the target object is determined, the optimization suggestion related to the target object or the target region can be further generated based on the action event. For example, in the case that the target region is a shopping mall and the target object is a customer, the action event of the customer, such as the entering and leaving event, determined by the above steps can be used to provide the management optimization suggestion for the management party of the shopping mall. For example, the display adjustment suggestion of the goods on the floor can be generated according to the entering and leaving trajectory of the customer, so as to set the goods which the customer hopes to purchase at the entrance and exit position where the user has a longer entering and leaving time or stays, and the like.

[0071] The image processing method provided in the embodiments of the present application determines the action event of the target object in the target region based on the trajectory information and the fitting curve, which not only greatly improves the efficiency and the accuracy of determination, but also provides a more accurate reference for the management of the target region or the service of the service party providing services for the target object, and further can adjust the subsequent display or improve the subsequent processing by displaying the trajectory line to the user and receiving the instruction of the user. In addition, the optimization suggestion can also be directly generated for the user based on the generated action event. In addition, the spline curve of the trajectory line can be obtained based on the fitting of the trajectory information in the embodiments of the present application, so that the spline curve is used as a constraint condition for fitting, which not only reduces the action trajectory to be processed, but also greatly reduces the calculation amount of fitting and improves the accuracy of the fitting result through the constraint of the spline curve.

[0072] Embodiment Four

[0073] Figure 4 The structure schematic diagram of the image processing device embodiment provided in the present application can be used to execute the method steps shown in Figure 2 and Figure 3 . As shown in Figure 4 , the image processing device can include a position information determination module 41, a trajectory information generation module 42, a fitting module 43 and an action event determination module 44.

[0074] The position information determination module 41 can be used to determine the position information of at least one target object in the target region according to a plurality of images of the target region.

[0075] In the embodiments of the present application, the position information of at least one target object in the region can be determined by image recognition from a series of images obtained for a pre-set target region. For example, in the scenario shown in Figure 1 , three customers are identified in the target region 1, so the position of the three customers in the images of the target region can be determined by the position information determination module 41. For example, the scenario shown in Figure 1 is the case of a shopping mall, and the three customers move in the shopping mall to view goods or perform shopping activities and the like within a certain time period, so the three customers can have a plurality of position points in the series of images of the region, and thus the position point information of each customer at different time points can be determined respectively by performing image recognition technology such as person recognition on the images.

[0076] The trajectory information generation module 42 can be used to generate the trajectory information of at least one target object according to the position information.

[0077] After the position information determination module 41 determines the position information of each customer, the trajectory information generation module 42 can generate trajectory information for each target object, i.e., customer, according to the position information. For example, the trajectory information generation module 42 can select a series of position points of a customer in images with predetermined time intervals in a high-frequency sampling manner to generate trajectory information, e.g., a trajectory line, of the customer. Note that since the action of a customer in a shopping mall usually has randomness, i.e., the customer does not move along a certain route, but can walk back and forth between two or several position points as shown in the middle of the figure, the trajectory information generation module 42 can obtain irregular or even somewhat chaotic lines by drawing a trajectory line by selecting position points in images with predetermined time intervals. Figure 1

[0078] The fitting module 43 can be configured to perform fitting processing on the trajectory information of each target object to generate a fitting curve of the target object.

[0079] Therefore, after the position information determination module 41 obtains the trajectory information of the target object, the fitting module 43 can generate a fitting curve according to the trajectory information of the target object recognized in the target region by fitting processing. In the embodiments of the present application, since the position information determination module 41 can usually recognize the position of each target object, i.e., customer, in the image, the trajectory information generation module 42 can accordingly obtain the trajectory of each customer, e.g., the trajectory information generation module 42 finally obtains a relatively chaotic trajectory composed of many trajectories. Such chaotic trajectories can only be manually assisted by line calibration based on vision and experience to determine the action, e.g., entry and exit behavior, of the customer in the prior art. Therefore, in the embodiments of the present application, the fitting module 43 can fit the chaotic multiple trajectories formed to a smaller number of fitting curves, e.g., a spline curve. For example, in the embodiments of the present application, trajectories with small intervals, i.e., very close to each other, can be fitted to generate a spline curve, or the trajectories of multiple customers, e.g., walking together, can be fitted to generate a trajectory to represent the moving trajectory of the group of customers, so that the computational amount of subsequent processing, e.g., determination processing, can be greatly reduced, and the processing efficiency can be improved.

[0080] In the embodiments of the present application, the fitting module 43 can be further configured to: group the trajectory information of at least one target object by using a clustering algorithm to generate multiple groups of trajectory information;

[0081] perform fitting processing on the multiple groups of trajectory information to obtain a spline curve of a trajectory line corresponding to the trajectory information;

[0082] generate a fitting curve by taking the multiple groups of trajectory information as training data and taking the spline curve as a constraint condition.

[0083] ​In the embodiments of the present application, the fitting module 43 can generate a trajectory line according to the directions of the trajectories generated by the trajectory information generation module 42 through, for example, an unsupervised clustering algorithm. For example, trajectories 1 and 2 are trajectories that converge upward, and trajectory 3 is a trajectory that converges downward, so trajectories 1 and 2 can be grouped together to generate a trajectory line, for example, to represent a concentrated passage of customers in the target area, and the direction of the passage can be determined in the direction of travel of trajectories 1 and 2, for example, left, right, or the opposite direction. For example, if trajectories 1 and 2 are in the same direction of travel, the direction of the passage can be determined to be unidirectional toward the direction of travel, and if trajectories 1 and 2 are in different directions, the direction of the passage can be determined to be bidirectional.

[0084] Therefore, on the basis of generating a plurality of groups of trajectory information, the plurality of groups of trajectory information can be fitted to obtain a spline curve of the trajectory line corresponding to the group of trajectory information. For example, in the embodiments of the present application, the grouped trajectory lines can be smoothed, i.e., spline smoothing, to obtain a spline curve of the trajectory line. The fitting module 43 can perform fitting processing using the generated grouped trajectory information as training data and using the spline curve as a constraint condition. For example, in the embodiments of the present application, a loss function can be designed according to the category of the auxiliary line to be calibrated, i.e., a loss model is constructed, and the loss model is trained by using the grouped trajectory lines determined by grouping to reflect the action characteristics of, for example, the passage passage in the target area, to ensure that the trajectory information obtained is input into the model to achieve the maximum data hit rate. For example, when calibrating the access auxiliary line, the auxiliary line category can be set to the access auxiliary line first, then the grouped trajectory information is used as training data, the spline curve is used as a constraint equation, and the loss function is used to fit the auxiliary line, and the training data is used to check whether the auxiliary line fitted in the current round can satisfy the determination that all the trajectory information is determined to be an access event, and if the trajectory information determined to be an access event is not enough, the fitting processing can be repeated until the determined auxiliary line meets the accuracy requirement. Thus, the determined auxiliary line is determined as the fitting curve.

[0085] The action event determination module 44 can be used to determine the action event of the target object according to the fitting curve.

[0086] In the embodiments of the present application, after the fitting module 43 obtains the fitting curve based on the trajectories of the target objects in the target region, the action event determination module 44 can determine the action event of the target objects according to the fitting curve. For example, since the fitting module 43 can process the trajectory lines of the trajectory information generation module 42 into a smaller number of fitting curves through the fitting processing, the action event determination module 44 can determine the access channel of the customer in the mall according to the trend and concentration degree of the fitting curves, and further determine the access behavior of the customer to the mall.

[0087] In addition, in the embodiments of the present application, the action event determination module 44 can be further used to:

[0088] determine the starting point or the ending point of the trajectory line according to the starting point or the ending point information in the trajectory information and the position information of the target objects;

[0089] obtain a fitting region according to the fitting curve;

[0090] determine the action event of the target objects in the target region according to the starting point or the ending point of the target objects appearing or leaving the fitting region.

[0091] In the embodiments of the present application, after the fitting module 43 fits the plurality of trajectory information into a smaller number of fitting curves, since such fitting curves can generally more accurately display the statistical trajectory of the customers who can walk in the region, the fitting region can be further determined according to the fitting curve. For example, the starting point or the ending point in the trajectory line can be determined according to the starting point or the ending point information in the trajectory information generated by the trajectory information generation module 42 and the position information obtained by the position information determination module 41, and the fitting region can be constructed based on the fitting curve around or passing through the starting point or the ending point.

[0092] After the fitting region is determined, the action event of the target objects, i.e., the access behavior of the customers to the target region, can be determined according to the starting point or the ending point of the position information of the target objects appearing or leaving the determined fitting region. Therefore, the access behavior of the customers can be determined with the region as a judgment range.

[0093] In addition, after the fitting region is determined, the judgment of the action event can also be similarly performed with the fitting region as a kind of visual field boundary. For example, the boundary of the determined fitting region can be determined as the visual field boundary, and the action event of the customers accessing the region can be determined according to the starting point or the ending point of the target objects entering or leaving the visual field boundary.

[0094] The image processing apparatus provided in the embodiments of the present application determines the position information of the target object in the target region and further generates trajectory information by recognizing the image of the target region, so as to determine the action event of the target object according to the fitting curve generated by fitting the fitting curve of the target object, and therefore, the embodiments of the present application can utilize the fitting curve based on the trajectory information to perform the processing such as the determination of the action event of the target object (for example, a customer) and the like, thereby eliminating the problem in the prior art that the auxiliary line needs to be manually calibrated based on the trajectory information, greatly improving the efficiency and the accuracy of the determination, and by determining the action event of the target object in the target region based on the trajectory information and the fitting curve, not only the efficiency and the accuracy of the determination are greatly improved, but also more accurate references are provided for the management of the target region or the service provider providing services for the target object, and further, the subsequent display or the subsequent processing can be adjusted or improved by displaying the trajectory line to the user and receiving the instruction of the user. In addition, the user can be directly provided with optimization suggestions based on the generated action event. In addition, the embodiments of the present application can obtain the spline curve of the trajectory line based on the fitting of the trajectory information, so as to use the spline curve as a constraint condition for fitting, not only the action trajectory to be processed is reduced, but also the calculation amount of the fitting is greatly reduced by the constraint of the spline curve, and the accuracy of the fitting result is improved.

[0095] Embodiment five

[0096] The internal functions and structures of the image processing apparatus are described above, and the apparatus can be implemented as an electronic device. Figure 5 The structural schematic diagram of the electronic device provided in the embodiments of the present application is shown in FIG. 1. Figure 5 As shown in FIG. 1, the electronic device includes a memory 51 and a processor 52.

[0097] The memory 51 is configured to store programs. In addition to the programs described above, the memory 51 can be configured to store various other data to support the operation on the electronic device. Examples of the data include instructions of any application program or method for operating on the electronic device, contact data, phonebook data, messages, pictures, videos, and the like.

[0098] The memory 51 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk or an optical disk.

[0099] The processor 52, which is not limited to a central processing unit (CPU), can also be a graphics processing unit (GPU), a field-programmable gate array (FPGA), an embedded neural processing unit (NPU), an artificial intelligence (AI) chip, or the like. The processor 52 is coupled to the memory 51 and executes a program stored in the memory 51, which, when executed, performs the image processing method of embodiments two and three described above.

[0100] Further, as shown in Figure 5 , the electronic device can further include a communication component 53, a power component 54, an audio component 55, a display 56, and other components. Figure 5 Some components are only schematically shown in the electronic device, and it does not mean that the electronic device only includes Figure 5 the components shown.

[0101] The communication component 53 is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 3G, 4G, or 5G, or a combination thereof. In an example embodiment, the communication component 53 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 53 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0102] The power component 54 provides power to various components of the electronic device. The power component 54 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the electronic device.

[0103] The audio component 55 is configured to output and / or input audio signals. For example, the audio component 55 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in the memory 51 or transmitted via the communication component 53. In some embodiments, the audio component 55 also includes a speaker for outputting audio signals.

[0104] The display 56 includes a screen, which can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect a duration and a pressure related to the touching or sliding action.

[0105] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program, when executed, performs steps including the above-mentioned method embodiments; and the foregoing storage medium includes ROM, RAM, magnetic disk or optical disk and various media that can store program codes.

[0106] Finally, it should be noted that the technical solutions of the present application only involve the analysis of the overall motion trajectory of the target object such as the customer, and do not involve the collection of the action trajectory and position of a specific particular target object individual, nor the privacy of the target object itself. In specific applications, if the technical solutions of the present application involve the privacy information of the target object itself, the collection of the privacy information such as the location information of the target object will be carried out on the premise of obtaining the consent of the target object.

[0107] In addition, the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, comprising: determining position information of at least one target object in a target area according to a plurality of images of the target area; generating trajectory information of the at least one target object according to the position information; grouping the trajectory information of the at least one target object by using a clustering algorithm to generate a plurality of groups of trajectory information; performing fitting processing on the plurality of groups of trajectory information to obtain a spline curve of a trajectory line corresponding to the trajectory information; generating a fitting curve by taking the plurality of groups of trajectory information as training data and taking the spline curve as a constraint condition; and determining an action event of the target object according to the fitting curve.

2. The image processing method of claim 1, wherein, The determining of the action event of the target object according to the fitting curve comprises: determining the action event of the target object according to the trajectory information and the fitting curve.

3. The image processing method of claim 2, wherein, The generating of the trajectory information of the at least one target object according to the position information comprises: generating a trajectory line of the target object in a high-frequency sampling manner according to the position information.

4. The image processing method of claim 3, wherein, The determining of the action event of the target object according to the trajectory information and the fitting curve comprises: determining a starting point or an ending point of the trajectory line of the target object according to starting point or ending point information in the trajectory information and the position information of the target object; obtaining a fitting area according to the fitting curve; regarding the appearance or departure of the starting point or the ending point of the target object in the fitting area as an action event of the target object in the target area.

5. The image processing method of claim 3, wherein, The determining of the action event of the target object according to the trajectory information and the fitting curve comprises: determining at least one fitting area according to a concentration degree of the fitting curve; constructing a visual field boundary based on a boundary of the at least one fitting area to determine entering and / or leaving of the target object into or out of the visual field boundary as an action event of the target object.

6. The image processing method of claim 1, further comprising: generating an optimization suggestion related to the target object or the target area according to the determined action event.

7. The image processing method of claim 6, wherein, The optimization suggestion comprises an adjustment suggestion of a commodity placement position in the target area.

8. The image processing method of claim 3, further comprising: displaying the generated trajectory line in a visualized form; receiving an instruction of a user for the generated trajectory line, wherein the instruction at least comprises an instruction related to the visualized display of the trajectory line; and displaying, according to the received instruction, a visualized display of the trajectory line adjusted according to the instruction related to the visualized display of the trajectory line to the user.

9. An image processing apparatus, comprising: a position information determination module configured to determine position information of at least one target object in a target area according to a plurality of images of the target area; a trajectory information generation module configured to generate trajectory information of the at least one target object according to the position information; a fitting module configured to group the trajectory information of the at least one target object by using a clustering algorithm to generate a plurality of groups of trajectory information; performing fitting processing on the plurality of groups of trajectory information to obtain a spline curve of a trajectory line corresponding to the trajectory information; and generating a fitting curve based on the multiple sets of trajectory information as training data and based on the spline curve as a constraint condition; an action event determination module configured to determine an action event of the target object based on the fitting curve.

10. The image processing apparatus according to claim 9, wherein The action event determination module is further configured to: determine action information of the target object based on the trajectory information and the fitting curve.

11. The image processing apparatus according to claim 10, wherein The trajectory information generation module is further configured to: generate a trajectory line of the target object in a high-frequency sampling manner based on the position information.

12. The image processing apparatus according to claim 11, wherein The action event determination module is further configured to: determine a starting point or an ending point of the trajectory line of the target object based on starting point or ending point information in the trajectory information of the target object and the position information of the target object; obtain a fitting region based on the fitting curve; determine an action event of the target object based on the starting point or the ending point of the target object appearing in or leaving the fitting region as the target object in the target region.

13. The image processing apparatus according to claim 11, wherein The action event determination module is further configured to: determine at least one fitting region based on a concentration degree of the fitting curve; construct a visual field boundary based on a boundary of the at least one fitting region to determine entering and / or leaving the visual field boundary of the target object as action information of the target object.

14. The image processing apparatus of claim 9, further comprising: a suggestion generation module configured to generate an optimization suggestion related to the target object or the target region based on the determined action event.

15. The image processing apparatus according to claim 14, wherein The optimization suggestion comprises an adjustment suggestion of a commodity placement position in the target region.

16. The image processing apparatus of claim 11, further comprising: a display module configured to display the generated trajectory line in a visualized form; an instruction receiving module configured to receive an instruction of a user on the generated trajectory line, wherein the instruction at least comprises an instruction related to the visualized display of the trajectory line; wherein the display module is further configured to show the user the visualized display of the trajectory line adjusted based on the instruction related to the visualized display of the trajectory line according to the instruction received by the instruction receiving module.

17. An electronic device, comprising: a memory configured to store a program; a processor configured to run the program stored in the memory, and the program, when running, performs the image processing method of any one of claims 1 to 8.

18. A computer readable storage medium having stored thereon a computer program, which can be executed by a processor, wherein, The program, when executed by the processor, implements the image processing method of any one of claims 1 to 8. The program, when executed by the processor, implements the image processing method of any one of claims 1 to 8.

Images