Image processing method and system, medium and electronic equipment
Through multimodal large language model and deep learning technology, combined with pedestrian re-identification network, the in-depth problem of pedestrian behavior analysis is solved, and in-depth analysis of pedestrian behavior and automation of consumption preference analysis is realized, and work costs are reduced.
Patent Information
- Application Number
- CN202410966071.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art is difficult to effectively analyze and understand the behavior of pedestrians in images or video sequences, especially the information obtained during pedestrian re-identification process, which leads to the analysis of consumption preferences relying on human judgment and increasing work costs.
Through multimodal large language model and deep learning technology, pedestrian recognition network, pedestrian images are extracted and trajectory optimization are obtained, pedestrian trajectory and basic information are obtained, behavior analysis is used for large language models, and information is integrated to achieve in-depth pedestrian behavior. analyze.
In-depth analysis and reasoning of pedestrian behavior is realized, labor costs are reduced, and the accuracy and automation of consumption preference analysis are improved, and the energy of data collection and labeling training for specific tasks is saved.
Smart Images

Figure CN118942112A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of artificial intelligence, and in particular, relates to an image processing method, system, medium and electronic device. Background Art
[0002] Person Re-identification (ReID), also known as pedestrian re-identification, is a new technology emerging in the field of intelligent video analysis. Person Re-identification uses computer vision technology to determine whether there is a specific pedestrian in an image or video sequence. In the process of pedestrian re-identification, a large amount of information about pedestrians will be obtained. How to use this information to conduct in-depth analysis of pedestrian behavior has become one of the technical problems that relevant technicians need to solve urgently. Summary of the invention
[0003] The purpose of this application is to provide an image processing method for in-depth analysis and reasoning of pedestrian behavior.
[0004] In a first aspect, the present application provides an image processing method, which includes: performing pedestrian re-identification on pedestrian images collected by cameras at different positions to obtain pedestrian trajectories; selecting a target image from the pedestrian image according to the pedestrian trajectory, and inputting the target image into a multimodal large language model to obtain a pedestrian state description; inputting the pedestrian state description into the large language model to obtain pedestrian behavior; inputting the pedestrian image into the multimodal large language model to obtain pedestrian basic information; inputting the pedestrian trajectory, the pedestrian behavior and the pedestrian basic information into the large language model to obtain pedestrian behavior analysis results.
[0005] In an implementation of the first aspect, pedestrian re-identification is performed on pedestrian images captured by cameras at different positions, and the process of obtaining pedestrian trajectories includes: using a multi-granularity pedestrian re-identification network to perform feature extraction processing on the pedestrian images to obtain features of the pedestrian images; performing cosine similarity calculation on the features of the pedestrian images to obtain cosine similarity; and obtaining the pedestrian trajectory based on a matching relationship between the cosine similarity and a similarity threshold.
[0006] In an implementation of the first aspect, pedestrian images captured by cameras at different positions are subjected to pedestrian re-identification, and the process of obtaining pedestrian trajectories also includes: logically optimizing the pedestrian trajectories according to the topology of each point in the pedestrian scene and the pedestrian capture time of the cameras at different positions.
[0007] In an implementation of the first aspect, the multi-granularity pedestrian re-identification network includes a four-layer structure, and the third layer of the multi-granularity pedestrian re-identification network includes three branches. The first branch is used to output global features, the second branch is used to output global features and two bisection features, and the third branch is used to output global features and three trisection features. The features output by the first branch, the second branch, and the third branch are output to the fully connected layer for dimensionality reduction after passing through the fourth layer, and the features after dimensionality reduction are spliced to obtain the final features of pedestrian re-identification.
[0008] In an implementation of the first aspect, selecting the target image from the pedestrian image according to the pedestrian trajectory includes: selecting a number of pedestrian entry images and / or exit images from the pedestrian image as the target image according to the pedestrian trajectory.
[0009] In an implementation of the first aspect, inputting the pedestrian state description into a large language model to obtain pedestrian behavior includes: generating a text prompt based on the pedestrian state description; and inputting the text prompt into the large language model to obtain the pedestrian behavior.
[0010] In an implementation of the first aspect, the target image is input into a large multimodal language model to obtain a pedestrian status description, including: inputting the target image and a corresponding text prompt into the large multimodal language model to obtain the pedestrian status description; the text prompt is used to guide the multimodal language model to output relevant information about the pedestrian status description.
[0011] In a second aspect, the present application provides an image processing system, which includes: a pedestrian trajectory acquisition module, which is used to perform pedestrian re-identification on pedestrian images collected by cameras at different positions to obtain pedestrian trajectories; a pedestrian state description module, which is used to select a target image from the pedestrian image according to the pedestrian trajectory, and input the target image into a multimodal large language model to obtain a pedestrian state description; a pedestrian behavior acquisition module, which is used to input the pedestrian state description into the large language model to obtain pedestrian behavior; a pedestrian basic information acquisition module, which is used to input the pedestrian image into the multimodal large language model to obtain pedestrian basic information; and an analysis module, which is used to input the pedestrian trajectory, the pedestrian behavior and the pedestrian basic information into the large language model to obtain pedestrian behavior analysis results.
[0012] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-described image processing methods.
[0013] In a fourth aspect, the present application provides an electronic device, comprising: a memory storing a computer program; and a processor communicatively connected to the memory, for executing the above-mentioned image processing method when the computer program is called.
[0014] As described above, the image processing method, system, medium and electronic device described in the present application have the following beneficial effects:
[0015] The image processing method provided in the embodiment of the present application can perform pedestrian re-identification on pedestrian images to obtain pedestrian trajectories, obtain pedestrian behavior through a large language model, and obtain pedestrian basic information through a multimodal large language model, integrate the above-mentioned pedestrian trajectory, pedestrian behavior and pedestrian basic information into the large language model to finally obtain the analysis result of the pedestrian behavior, thereby realizing in-depth analysis and reasoning of pedestrian behavior. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 Shown is a schematic diagram of an application scenario of the image processing method described in an embodiment of the present application.
[0017] Figure 2 Shown is a flowchart of the image processing method described in an embodiment of the present application.
[0018] Figure 3 Shown is a schematic diagram of the process of obtaining pedestrian trajectories described in an embodiment of the present application.
[0019] Figure 4 Shown is a schematic diagram of the structure of the multi-granularity pedestrian re-identification network described in an embodiment of the present application.
[0020] Figure 5 Shown is a schematic diagram of the process of obtaining pedestrian behavior as described in an embodiment of the present application.
[0021] Figure 6 Shown is a schematic diagram of the process of pedestrian shopping analysis described in an embodiment of the present application.
[0022] Figure 7 Shown is a structural schematic diagram of the image processing system described in an embodiment of the present application.
[0023] Figure 8 Shown is a schematic diagram of the structure of an electronic device described in an embodiment of the present application.
[0024] Component number description
[0025] 4 Image Processing System
[0026] 41 Pedestrian trajectory acquisition module
[0027] 42 Pedestrian status description module
[0028] 43 Pedestrian Behavior Acquisition Module
[0029] 44 Pedestrian basic information acquisition module
[0030] 45 Analysis Module
[0031] 5 Electronic devices
[0032] 51 Memory
[0033] 52 Processors
[0034] Steps S1 to S5
[0035] Steps S11 to S14
[0036] Steps S31 to S32 DETAILED DESCRIPTION
[0037] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.
[0038] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.
[0039] Person re-identification uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. Person re-identification can be used in a variety of application scenarios. For ease of explanation, the following will take the application of person re-identification in a pedestrian shopping scenario as an example, but this application is not limited to this.
[0040] When applying pedestrian re-identification to pedestrian shopping scenarios, the pedestrian's walking trajectory in the mall can be obtained by matching the similarity of the pedestrian's appearance and clothing features. However, these trajectory information has not been fully and effectively analyzed, and usually requires human thinking and judgment to analyze the person's consumption preferences.
[0041] In addition, the consumption behavior of pedestrians during shopping, such as extra shopping bags after shopping, requires the use of object recognition visual models to judge. If it involves judging the gender and age of pedestrians, another deep learning model is needed to judge, which increases the work cost.
[0042] At least to address the above-mentioned problems, the following embodiments of the present application provide an image processing method. Figure 1 The following is a schematic diagram of an application scenario of the image processing method provided by this application. Figure 1 As shown, cameras set up at different locations in the mall can collect images of pedestrians' appearance. Electronic devices, such as local hosts or remote servers, can use these images to dynamically track user trajectories in real time, understand the user's wandering behavior trajectory in the mall, and then understand the user's interests by analyzing the behavior trajectory, so as to optimize the user's shopping experience.
[0043] Figure 2 The flowchart of the image processing method provided by the present application in a specific embodiment is shown. The image processing method can be applied to the aforementioned electronic device. Figure 2 As shown, the image processing method includes:
[0044] S1, perform pedestrian re-identification on pedestrian images collected by cameras at different positions to obtain pedestrian trajectories.
[0045] Specifically, the cameras are arranged at different positions in the mall, and can capture pedestrians at different positions in the mall. The captured pedestrian images are input into the pedestrian re-identification module to obtain the pedestrian's wandering trajectory in the mall.
[0046] S2, selecting a target image from the pedestrian images according to the pedestrian trajectory, and inputting the target image into a multimodal large language model to obtain a pedestrian state description.
[0047] Specifically, the target image refers to an image that is convenient for obtaining a description of the pedestrian's state. For example, for a store that a pedestrian has entered during his or her wandering trajectory, multiple images taken before and after the pedestrian enters or leaves the store can be selected as the target image. Among the multiple images taken before and after the pedestrian enters or leaves the store, if there is an image of the pedestrian's orientation information, the pedestrian's front view and / or an image close to the time of entering the store can be selected.
[0048] It should be noted that the pedestrian orientation information includes two ways of obtaining, one is recognition through a deep learning model, and the other is obtained through a large multimodal model, but the present application is not limited to this.
[0049] S3, input the pedestrian state description into a large language model to obtain pedestrian behavior. Among them, the large language model is a model built based on deep learning and natural language processing technology, which can understand and generate complex human language. Combining pedestrian trajectory data and other relevant information (such as store types in a shopping mall) and inputting them into a large language model can achieve more efficient and in-depth data analysis, thereby obtaining accurate pedestrian behavior. In some implementations, the large language model can be LLaMA (Large Language Model Meta AI, Meta Artificial Intelligence Large Language Model) and the like.
[0050] S4, input the pedestrian image into the multimodal large language model (MLLM) to obtain the basic information of the pedestrian. The multimodal large language model can flexibly process pedestrian attributes, such as age, gender and other information. The multimodal large language model can simultaneously process and understand data from multiple different modes, such as text, images, audio and video. It is not limited to the understanding and processing of a single mode (such as only text or only image), but can integrate different types of information to provide more comprehensive and in-depth data analysis. If images and text are involved, such as visual question and answer tasks, the multimodal large language model has better generalization ability than the model based on deep learning. The multimodal large language model is trained on a larger and more diverse data machine, can better handle multiple different types of visual question and answer tasks and has more advanced reasoning capabilities, thereby obtaining accurate basic information about pedestrians.
[0051] S5, inputting the pedestrian trajectory, the pedestrian behavior and the pedestrian basic information into the large language model to obtain pedestrian behavior analysis results.
[0052] According to the above description, it can be seen that the image processing method provided in the embodiment of the present application can perform pedestrian re-identification on pedestrian images to obtain pedestrian trajectories, obtain pedestrian behavior through a large language model, and obtain pedestrian basic information through a multimodal large language model. The above-mentioned pedestrian trajectory, pedestrian behavior and pedestrian basic information are integrated and input into the large language model to finally obtain the analysis results of pedestrian behavior.
[0053] Figure 3 The figure shows a flow chart of obtaining pedestrian trajectories in one embodiment of the present application. Figure 3 As shown in FIG. 1 , the process of performing pedestrian re-identification on pedestrian images collected by cameras at different positions and obtaining pedestrian trajectories includes:
[0054] S11, using a multi-granularity pedestrian re-identification network to perform feature processing on the pedestrian image to obtain features of the pedestrian image.
[0055] The multi-granularity pedestrian re-identification network is used to continuously track the overall features of pedestrians in pedestrian images across cameras, and the overall features of pedestrians include characteristic information such as pedestrian clothing, body shape, hairstyle and posture. The multi-granularity pedestrian re-identification network is a multi-branch deep network, an end-to-end feature learning strategy that combines global information with local information of each granularity, and obtains multi-granularity local feature representation by dividing the pedestrian image into multiple strips and changing the number of strips of different local branches.
[0056] S12, performing cosine similarity calculation on the features of the pedestrian image to obtain cosine similarity.
[0057] The cosine similarity is an indicator that measures the similarity between two feature vectors, and is expressed as the matrix inner product of the two features normalized by the L2 norm. In some implementations, the calculation formula of the cosine similarity is as follows:
[0058]
[0059] Wherein, A and B represent two eigenvectors of the pedestrian image respectively, A·B represents the dot product of the two eigenvectors, and ||A|| and ||B|| represent the modulus lengths of the eigenvector A and the eigenvector B respectively.
[0060] S13, obtaining the pedestrian trajectory according to the matching relationship between the cosine similarity and a similarity threshold. The similarity threshold can be set according to actual needs or experience.
[0061] Specifically, the features of the pedestrian image are extracted through the multi-granularity pedestrian re-identification network, and the cosine similarity between two features is calculated. When the cosine similarity is greater than a similarity threshold, the two pedestrian images are considered to be the same pedestrian image.
[0062] Please continue reading Figure 3 , pedestrian re-identification is performed on pedestrian images collected by cameras at different positions, and the process of obtaining pedestrian trajectories may also include:
[0063] S14, performing logical optimization on the pedestrian trajectory according to the topology of each point in the pedestrian scene and the pedestrian capture time of the cameras at different positions.
[0064] Specifically, the search range can be further restricted according to the topology of each point in the mall and the time when the pedestrians were captured, so as to obtain a more reasonable and accurate trajectory of pedestrians in the mall.
[0065] Figure 4 The diagram shows a schematic diagram of the structure of a multi-granularity person re-identification network in one embodiment of the present application. Figure 4As shown, the multi-granularity pedestrian re-identification network includes a four-layer structure. The first two layers of the multi-granularity pedestrian re-identification network are used to process the image to obtain the global feature map of the pedestrian image, and the third layer of the multi-granularity pedestrian re-identification network includes three branches. The first branch is used to output global features, the second branch is used to output global features and two bisection features, and the third branch is used to output global features and three trisection features. The features output by the first branch, the second branch, and the third branch are output to the fully connected layer after passing through the fourth layer for dimensionality reduction. The features after dimensionality reduction are spliced to obtain the final features of pedestrian re-identification. The pedestrian trajectory is obtained based on the final features of pedestrian re-identification.
[0066] Exemplarily, the multi-granularity pedestrian re-identification network can use Resnet50 as the trunk network to extract pedestrian image information, but the present application is not limited to this.
[0067] The Resnet50 includes 4 layers, each of which is composed of a different number of convolutional layers. Among them, the area of the feature map output by the layer is one-fourth of the area of the input pedestrian image. Assuming that the width and height of the pedestrian image input to Resnet50 are 192*384, the pedestrian image is iterated in a decreasing manner in the first two layers of Resnet50, and the width and height of the feature map output by layer3 of Resnet50 become 12*24. The two bisection features of the second branch in layer3 can be represented as the upper and lower body of the pedestrian, and the three trisection features of the third branch can be represented as dividing the pedestrian into three parts: upper, middle and lower. The structure of the last layer4 of the Resnet50 is copied to 8, and the 8 blocks of the three branches of layer3 of the Resnet50 are respectively input into the copied layer4, and output to the fully connected layer fc to reduce the dimension to 256, and the 8 groups of features are spliced to obtain the final features.
[0068] Furthermore, the features after dimensionality reduction of the fully connected layer are concatenated to obtain the final features of pedestrian re-identification, which can be expressed as: concatenating 3 256-dimensional global feature vectors and 5 256-dimensional local feature vectors to obtain a 2048-dimensional vector as the feature representation of the pedestrian.
[0069] In one embodiment of the present application, selecting a target image from the pedestrian image according to the pedestrian trajectory includes: selecting a number of pedestrian entry images and / or exit images from the pedestrian image as the target image according to the pedestrian trajectory.
[0070] Figure 5 The figure shows a flow chart of obtaining pedestrian behavior in one embodiment of the present application. Figure 5 As shown, the pedestrian state description is input into the large language model to obtain the pedestrian behavior including:
[0071] S31, generating a first round of text prompts according to the pedestrian status description.
[0072] S32: input the first round of text prompts into the large language model to obtain the pedestrian behavior.
[0073] Exemplarily, the first round of text prompts is, for example: "The following list describes the same pedestrian in different stores, the three text descriptions of the pedestrian before entering the store and the three text descriptions of the pedestrian after leaving the store [Store: store1, entering the store: [text1, text2, text3], leaving the store: [text1, text2, text3]], [Store: store2, entering...], please infer whether the pedestrian has made any purchases in a store based on the description of the pedestrian's status entering and leaving the store. For example, if the pedestrian has extra shopping bags, drinks or other goods in his hands, it can be considered that the pedestrian has made purchases in the store he entered earlier, but the extra goods in his hands need to be related to the items sold in the store."
[0074] In one embodiment of the present application, the pedestrian trajectory, the pedestrian behavior and the pedestrian basic information are input into the large language model, and the pedestrian behavior analysis result is obtained, including: the pedestrian trajectory, the pedestrian behavior and the pedestrian basic information and the second round of text prompts are input into the large language model, and the large language model can output the pedestrian behavior analysis result. Exemplarily, the second round of text prompts is, for example: "The store browsing information of pedestrians in the store is as follows: [[Store 1, entry time, exit time], [Store 2, entry time, exit time], ...]".
[0075] In some implementations, the large language model can use qwen vi , but this application is not limited to this.
[0076] In one embodiment of the present application, the target image is input into a multimodal large language model to obtain a pedestrian state description including:
[0077] The target image and the corresponding text prompt are input into the multimodal large language model to obtain the pedestrian status description. The text prompt is used to guide the multimodal language model to output relevant information related to the pedestrian status description. The pedestrian status description may be, for example, the color of the pedestrian's clothing, the type of clothing, and the accessories and items carried. The text prompt may be, for example: Please describe the color of the pedestrian's clothing, the type of clothing, and the accessories and items carried.
[0078] In some implementations, the multimodal large language model may use LLaVA or qwen-vl, but the present application is not limited thereto.
[0079] Figure 6 The following is a schematic diagram of the process of pedestrian shopping analysis in one embodiment of the present application. Figure 6 As shown in the figure, pedestrian shopping analysis includes two parts: the first part is pedestrian re-identification, and the second part is consumption analysis. Specifically, pedestrian images are captured by cameras set at different locations in the shopping mall, and the pedestrian images are input into the pedestrian re-identification module for pedestrian re-identification to obtain the pedestrian's wandering trajectory in the shopping mall, which includes the time and store where the pedestrian appears. Consumption analysis is performed on the pedestrian trajectory after pedestrian re-identification, and a sequence of pictures of pedestrians entering and leaving the store is obtained according to the pedestrian's wandering trajectory in the shopping mall, and the age and gender information of the pedestrian is obtained using a multimodal large language model. At the same time, the large language model is used to determine whether the pedestrian has purchased goods based on the difference in descriptions of pedestrians entering and leaving the store. Finally, the pedestrian's wandering trajectory in the shopping mall, the pedestrian's age and gender information, and the behavioral judgment of whether the pedestrian has purchased goods are integrated and input into the large language model to obtain the pedestrian's shopping analysis and consumption preference inference.
[0080] In this embodiment, the pedestrian's walking trajectory in the mall, the pedestrian's gender and age group, the attributes of the store, and whether the pedestrian has made a purchase after entering the store are integrated and input into a large language model to analyze and infer the pedestrian's shopping and consumption preferences. The pedestrian's gender and age group are inferred using a multimodal large language model. For the judgment of whether to shop, in addition to using a multimodal large language model, the multi-image trajectory of ReID is also used to solve the missed detection caused by occlusion of the view of some pictures, and the multi-image of ReID can be used to eliminate false detection. In this way, the analysis results of pedestrian shopping analysis and consumption preferences can be accurately obtained.
[0081] In summary, this application combines the pedestrian re-identification task, the reasoning of the multimodal large model's image-text interaction, and the reasoning of the large language model, so that the pedestrian trajectory results output by the pedestrian re-identification can obtain deeper information mining. Thanks to the flexibility and strong generalization ability of the multimodal large model's image-text interaction, it saves the manpower and energy of the complicated data collection and annotation training for specified tasks in the deep learning model, such as the judgment of pedestrian age and gender and the analysis of pedestrians' clothing and carrying items. In addition, through the description of pedestrians entering and leaving the store output by the multimodal large model combined with the analysis of the large language model, effective shopping information is extracted in a very simple way, reducing the cost of solving the task. Finally, through the combined interaction of multiple methods, the shopping and consumption preferences of pedestrians are more reasonably inferred and analyzed, and the analysis of pedestrian shopping behavior is automated, reducing labor costs.
[0082] The protection scope of the image processing method described in the embodiment of the present application is not limited to the execution order of the steps listed in the present embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.
[0083] An embodiment of the present application also provides an image processing system, which can implement the image processing method described in the present application. However, the implementation device of the image processing method described in the present application includes but is not limited to the structure of the image processing system listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.
[0084] Figure 7 The diagram is a schematic diagram of the structure of an image processing system in one embodiment of the present application. Figure 7 As shown, the image processing system 4 includes: a pedestrian trajectory acquisition module 41, a pedestrian state description module 42, a pedestrian behavior acquisition module 43, a pedestrian basic information acquisition module 44 and an analysis module 45. Among them, the pedestrian trajectory acquisition module 41 is used to perform pedestrian re-identification on pedestrian images collected by cameras at different positions to obtain pedestrian trajectories. The pedestrian state description module 42 is used to select a target image from the pedestrian image according to the pedestrian trajectory, input the target image into a multimodal large language model, and obtain a pedestrian state description. The pedestrian behavior acquisition module 43 is used to input the pedestrian state description into the large language model to obtain pedestrian behavior. The pedestrian basic information acquisition module 44 is used to input the pedestrian image into the multimodal large language model to obtain pedestrian basic information. The analysis module 45 is used to input the pedestrian trajectory, the pedestrian behavior and the pedestrian basic information into the large language model to obtain pedestrian behavior analysis results.
[0085] It should be noted that the pedestrian trajectory acquisition module 41, the pedestrian state description module 42, the pedestrian behavior acquisition module 43, the pedestrian basic information acquisition module 44 and the analysis module 45 in the image processing system 4 are Figure 2 Steps S1 to S5 in the image processing method shown correspond to each other and are not described in detail here.
[0086] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.
[0087] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.
[0088] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0089] The present application embodiment also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by a program to instruct the processor, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state hard disk (SSD)), etc.
[0090] The embodiment of the present application may also provide an electronic device. Figure 8 The diagram is a schematic diagram of the structure of an electronic device 5 in one embodiment of the present application. Figure 8 As shown, in this embodiment, the electronic device 5 includes a memory 51 and a processor 52. The memory 51 is used to store computer programs. In some possible implementations, the memory 51 may include: ROM, RAM, disk, USB flash drive, memory card or CD-ROM and other media that can store program codes.
[0091] In the embodiment of the present application, the memory 51 may include a computer system readable medium in the form of a volatile memory, such as a RAM and / or a cache memory. The electronic device 5 may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 51 may include at least one program product having a set (e.g., at least one) of program modules, which are configured to perform the functions of the various embodiments of the present application.
[0092] The processor 52 is connected to the memory 51 and is used to execute the computer program stored in the memory 51 so that the electronic device 5 executes the image processing method.
[0093] Exemplarily, the processor 52 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. In other embodiments, the processor 52 may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0094] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.
[0095] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.
Claims
1. An image processing method, characterized in that: The image processing method comprises: Perform pedestrian re-identification on pedestrian images collected by cameras at different locations to obtain pedestrian trajectories; Selecting a target image from the pedestrian image according to the pedestrian trajectory, inputting the target image into a multimodal large language model to obtain a pedestrian state description; Inputting the pedestrian state description into a large language model to obtain pedestrian behavior; Inputting the pedestrian image into the multimodal large language model to obtain basic information of the pedestrian; The pedestrian trajectory, the pedestrian behavior and the pedestrian basic information are input into the large language model to obtain a pedestrian behavior analysis result.
2. The image processing method according to claim 1, characterized in that: The process of re-identifying pedestrians from images of pedestrians collected by cameras at different locations and obtaining pedestrian trajectories includes: Performing feature extraction processing on the pedestrian image using a multi-granularity person re-identification network to obtain features of the pedestrian image; Performing cosine similarity calculation on the features of the pedestrian image to obtain cosine similarity; The pedestrian trajectory is obtained according to a matching relationship between the cosine similarity and a similarity threshold.
3. The image processing method according to claim 2, characterized in that: The process of re-identifying pedestrians from images of pedestrians collected by cameras at different locations and obtaining pedestrian trajectories also includes: The pedestrian trajectory is logically optimized according to the topology of each point in the pedestrian scene and the pedestrian capture time of the cameras at different positions.
4. The image processing method according to claim 2, characterized in that: The multi-granularity person re-identification network includes a four-layer structure. The third layer of the multi-granularity person re-identification network includes three branches. The first branch is used to output global features, the second branch is used to output global features and two bisection features, and the third branch is used to output global features and three trisection features. The features output by the first branch, the second branch and the third branch are output to the fully connected layer for dimensionality reduction after passing through the fourth layer. The features after dimensionality reduction are spliced to obtain the final features of pedestrian re-identification.
5. The image processing method according to claim 1, characterized in that: Selecting a target image from the pedestrian image according to the pedestrian trajectory includes: selecting a number of pedestrian entry-into-store images and / or store exit-into-store images from the pedestrian image according to the pedestrian trajectory as the target image.
6. The image processing method according to claim 1, characterized in that: Inputting the pedestrian state description into the large language model to obtain pedestrian behaviors includes: generating a text prompt according to the pedestrian status description; The text prompt is input into the large language model to obtain the pedestrian behavior.
7. The image processing method according to claim 1, characterized in that: The target image is input into a multimodal large language model to obtain a pedestrian state description including: The target image and the corresponding text prompt are input into the multimodal large language model to obtain the pedestrian status description; the text prompt is used to guide the multimodal language model to output relevant information about the pedestrian status description.
8. An image processing system, characterized in that: The image processing system comprises: The pedestrian trajectory acquisition module is used to perform pedestrian re-identification on pedestrian images collected by cameras at different locations to obtain pedestrian trajectories; A pedestrian state description module, used for selecting a target image from the pedestrian image according to the pedestrian trajectory, inputting the target image into a multimodal large language model, and obtaining a pedestrian state description; A pedestrian behavior acquisition module, used for inputting the pedestrian state description into a large language model to obtain pedestrian behavior; A pedestrian basic information acquisition module, used for inputting the pedestrian image into the multimodal large language model to obtain the pedestrian basic information; The analysis module is used to input the pedestrian trajectory, the pedestrian behavior and the pedestrian basic information into the large language model to obtain the pedestrian behavior analysis result.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 7 is implemented.
10. An electronic device, characterized in that: The electronic device comprises: A memory storing a computer program; A processor is communicatively connected to the memory, and executes the image processing method according to any one of claims 1 to 7 when calling the computer program.
Citation Information
Cited By
Information processing apparatus, information processing method, and information processing program
JP2026093200A
Information processing device, information processing method, and information processing program
JP7858026B1
system
JP7892094B1
