Passenger flow analysis method and system based on multi-modal large model, and server

Abstract

The invention provides a passenger flow analysis method and system based on a multi-modal large model and a server, and the method comprises the steps: obtaining video data in real time through a streaming media service, analyzing the video data in real time to generate a real-time analysis result, and transmitting the real-time analysis result to a message pool through a message transmission service, and generating answer information through a large language model according to the user question information, the interface information, the prompt information and the real-time analysis result, and sending the answer information to the client for display. The real-time analysis result is obtained based on the video data, the real-time performance and accuracy of the analysis result can be guaranteed, the answer information is generated through the large language model, high expansibility is achieved, and the diversified requirements of users are met.

Description

Technical Field

[0001] The present application relates to the technical field of passenger flow analysis, and in particular, to a passenger flow analysis method, system and server based on a multimodal large model. Background Art

[0002] With the rapid development of urbanization, the efficiency and safety of personnel flow in places where people gather have attracted much attention. How to reasonably and effectively evaluate the passenger flow congestion in places where people gather is of great significance to ensuring the comfortable travel of people.

[0003] There are two existing solutions for passenger flow calculation: one is to use mobile phone code detection equipment to collect mobile phone feature codes and perform statistics to generate passenger flow heat maps. The other is to collect personnel flow data through base stations and thermal imaging equipment.

[0004] However, the existing method of collecting mobile phone features is overly dependent on mobile phone signals, and the accuracy of passenger flow heat maps cannot be guaranteed. The acquisition base station and thermal imaging equipment can only collect personnel flow data, and cannot analyze and evaluate passenger flow status, which has poor scalability. Summary of the invention

[0005] The purpose of this application is to provide a passenger flow analysis method, system and server based on a multimodal large model to address the deficiencies in the above-mentioned prior art, so as to solve the problems of poor accuracy and poor scalability in the prior art.

[0006] To achieve the above objectives, the technical solutions adopted in this application are as follows:

[0007] In a first aspect, the present application provides a passenger flow analysis method based on a multimodal large model, which is applied to a passenger flow analysis system, wherein the passenger flow analysis system includes a server and a client, and the server is deployed with: a streaming media service, a pre-trained multimodal large model, a message transmission service, and a large language model, and the method includes:

[0008] Acquire video data of the target area to be analyzed in real time through the streaming media service;

[0009] Performing real-time analysis on the video data through the multimodal large model to generate real-time analysis results, wherein the real-time analysis results include: object position results, queuing detection results, retrograde detection results, and fall detection results;

[0010] Sending the real-time analysis result to the message pool of the server through the message transmission service;

[0011] The real-time analysis result is read from the message pool through the large language model, and answer information corresponding to the user question information is generated according to the user question information sent by the client, the preset interface information, the preset prompt information and the real-time analysis result, and the answer information is sent to the client for display through the message transmission service, the interface information includes parameter information of the analysis algorithm to be called by the large language model, and the prompt information includes the data format of the analysis algorithm.

[0012] Optionally, performing real-time analysis on the video data to generate real-time analysis results includes:

[0013] Identify the object in the video data, determine the initial position result, and convert the initial position result into the object position result, wherein the initial position result is the result under the acquisition coordinates, and the object position result is the coordinate under the two-dimensional drawing of the scene;

[0014] According to the video data, the number of people in the preset queuing area in the target area is counted, and the queuing detection result is generated according to the statistical result and the queuing threshold;

[0015] According to the video data, the orientation, travel direction and preset escalator direction of the personnel in the preset escalator area in the target area are judged, and the reverse detection result is generated according to the judgment result;

[0016] The posture of the person in the video data is identified, and the fall detection result is generated according to the identification result.

[0017] Optionally, converting the initial position result into the object position result comprises:

[0018] Obtaining initial positions corresponding to a plurality of preset position points in the initial position result;

[0019] Obtaining the target position corresponding to each of the preset position points;

[0020] Determining coordinate conversion information according to the initial position corresponding to each of the preset position points and the target position corresponding to each of the preset position points;

[0021] The initial position result is transformed according to the coordinate transformation information to obtain the object position result.

[0023] Optionally, the training process of the multimodal large model includes:

[0024] Inputting a first image sample into an initial image model and a first text sample into an initial text model, performing comparative training on the initial image model and the initial text model, and obtaining a target image model, a target text model, and a weight of the target text model;

[0025] Establishing an initial fine-tuning model based on the weights of the target text model, and inputting the first training sample into the initial fine-tuning model for training to obtain a target fine-tuning model;

[0026] The second training sample is respectively input into the target image model, the target text model and the target fine-tuning model, and the first prediction result, the second prediction result and the third prediction result are respectively output, the second training sample includes a second image sample, a second text sample and a prompt word, and the first prediction result, the second prediction result and the third prediction result are input into the decoder to obtain a fourth prediction result, and the target image model, the target text model, the target fine-tuning model and the decoder are iteratively corrected according to the first prediction result, the second prediction result, the third prediction result and the fourth prediction result to obtain the multimodal large model.

[0027] Optionally, the performing comparative training on the initial image model and the initial text model to obtain a target image model and the target text model includes:

[0028] Performing cross entropy loss calculation according to an actual result of the image model output by the initial image model and an actual result of the text model output by the initial text model to obtain loss information;

[0029] The initial image model and the initial text model are iteratively modified according to the loss information to obtain a target image model and the target text model.

[0030] Optionally, generating answer information corresponding to the user question information according to the user question information sent by the client, preset interface information, preset prompt information and the real-time analysis result includes:

[0031] Selecting a target interface from among multiple interfaces based on the user question information, the interface information, and the prompt information;

[0032] Calling the target interface, and executing the analysis algorithm by the target interface to obtain an intermediate result according to the real-time analysis result;

[0033] The answer information is extracted based on the user question information, the intermediate results and the real-time analysis results.

[0034] Optionally, the method further comprises:

[0035] The real-time analysis result is sent to the client for display via the message transmission service.

[0036] In a second aspect, the present application provides a passenger flow analysis system, which includes a server and a client, and the server is deployed with: a streaming media service, a pre-trained multimodal large model, a message transmission service, and a large language model:

[0037] The server is used to obtain video data of the target area to be analyzed in real time through the streaming media service;

[0038] The server is used to perform real-time analysis on the video data through the multimodal large model to generate real-time analysis results, wherein the real-time analysis results include: object position results, queuing detection results, retrograde detection results, and fall detection results;

[0039] The server is used to send the real-time analysis result to the message pool of the server through the message transmission service;

[0040] The server is used to read the real-time analysis result from the message pool through the large language model, and generate answer information corresponding to the user question information according to the user question information sent by the client, preset interface information, preset prompt information and the real-time analysis result, and send the answer information to the client for display through the message transmission service, the interface information includes parameter information of the analysis algorithm to be called by the large language model, and the prompt information includes the data format of the analysis algorithm.

[0041] Optionally, the server is used to identify the object in the video data, determine an initial position result, and convert the initial position result into the object position result, wherein the initial position result is a result under acquisition coordinates, and the object position result is a coordinate under a two-dimensional drawing of the scene;

[0042] According to the video data, the number of people in the preset queuing area in the target area is counted, and the queuing detection result is generated according to the statistical result and the queuing threshold;

[0043] According to the video data, the orientation, travel direction and preset escalator direction of the personnel in the preset escalator area in the target area are judged, and the reverse detection result is generated according to the judgment result;

[0044] The posture of the person in the video data is identified, and the fall detection result is generated according to the identification result.

[0045] Optionally, the server is used to obtain initial positions corresponding to a plurality of preset position points in the initial position result;

[0046] Obtaining the target position corresponding to each of the preset position points;

[0047] Determining coordinate conversion information according to the initial position corresponding to each of the preset position points and the target position corresponding to each of the preset position points;

[0048] The initial position result is transformed according to the coordinate transformation information to obtain the object position result.

[0049] Optionally, the server is used to input the first image sample into an initial image model, and input the first text sample into an initial text model, perform comparative training on the initial image model and the initial text model, and obtain a target image model, a target text model, and a weight of the target text model;

[0050] Establishing an initial fine-tuning model based on the weights of the target text model, and inputting the first training sample into the initial fine-tuning model for training to obtain a target fine-tuning model;

[0051] The second training sample is respectively input into the target image model, the target text model and the target fine-tuning model, and the first prediction result, the second prediction result and the third prediction result are respectively output, wherein the second training sample includes a second image sample, a second text sample and a prompt word, and the first prediction result, the second prediction result and the third prediction result are input into the decoder to obtain a fourth prediction result, and according to the first prediction result, the second prediction result, the third prediction result and the fourth prediction result, the target image model, the target text model, the target fine-tuning model and the decoder are iteratively corrected to obtain the multimodal large model.

[0052] Optionally, the server is used to perform cross entropy loss calculation according to an actual result of the image model output by the initial image model and an actual result of the text model output by the initial text model to obtain loss information;

[0053] The initial image model and the initial text model are iteratively modified according to the loss information to obtain a target image model and the target text model.

[0054] Optionally, the server is used to select a target interface from a plurality of interfaces based on the user question information, the interface information and the prompt information;

[0055] Calling the target interface, and executing the analysis algorithm by the target interface to obtain an intermediate result according to the real-time analysis result;

[0056] The answer information is extracted based on the user question information, the intermediate results and the real-time analysis results.

[0057] Optionally, the server is used to send the real-time analysis result to the client for display via the message transmission service.

[0058] In a third aspect, the present application provides a server comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the server is running, the processor executes the machine-readable instructions to perform the steps of the passenger flow analysis method based on a multimodal large model as described in the first aspect.

[0059] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are executed.

[0060] The beneficial effects of the present application are: first, video data is obtained in real time through a streaming media service, and a multimodal large model is used to perform real-time analysis on the video data to generate real-time analysis results. The real-time performance of the analysis result output can be improved by real-time analysis of video data, so that users can make corresponding actions to the personnel status of the current target area in a timely manner according to the real-time analysis results. In addition, the present embodiment analyzes the current passenger flow from multiple angles through video data, with high scalability and high accuracy. Next, the real-time analysis results are sent to the message pool through a message transmission service. Since the message transmission service is timely, the real-time analysis results can be stored in the message pool in a timely manner to be called for display or further analysis. The further analysis process is to read the real-time analysis results from the message pool through a large language model, and generate the answer information corresponding to the user's question information according to the user's question information, interface information, prompt information and real-time analysis results. The real-time analysis results are further analyzed by a large language model, thereby providing user analysis results and suggestions to meet the diverse actual needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 It is a flowchart of a passenger flow analysis method based on a multimodal large model provided in an embodiment of the present application;

[0063] Figure 2 It is a flowchart of a method for generating real-time analysis results provided by an embodiment of the present application;

[0064] Figure 3 It is a flowchart of a method for converting position results provided in an embodiment of the present application;

[0065] Figure 4 It is a schematic diagram of the relationship between the initial position and the target position of a preset position point provided in an embodiment of the present application;

[0066] Figure 5 It is a schematic diagram of a training process of a multimodal large model provided in an embodiment of the present application;

[0067] Figure 6 It is a flowchart of a method for generating answer information proposed in an embodiment of the present application;

[0068] Figure 7 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0070] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0071] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0072] In places with large passenger flow, such as rail transit and scenic spots, passenger flow analysis is very important. There are two existing passenger flow calculation solutions. One is to use mobile phone code detection equipment to collect mobile phone feature codes and perform statistics to generate a passenger flow heat map. However, this method of collecting mobile phone features is too dependent on mobile phone signals, and the accuracy of the passenger flow heat map cannot be guaranteed. The other is to collect personnel flow data through base stations and thermal imaging equipment, including telephone numbers, latitude and longitude of the collected equipment, and collection time, to generate a passenger flow heat map. However, this method cannot analyze and evaluate the passenger flow status and has poor scalability.

[0073] Based on this, the present application proposes a passenger flow analysis method based on a multimodal large model. The method is applied to a passenger flow analysis system, obtains video data in real time through a streaming media service, and analyzes the video data in real time through a multimodal large model to generate real-time analysis results. The real-time analysis results include object position results, queuing detection results, retrograde detection results, and fall detection results. The above-mentioned real-time analysis results are obtained based on video data, which can ensure the real-time and accuracy of the analysis results. Then the real-time analysis results are sent to the message pool of the server through the message transmission service, and finally the real-time analysis results are read from the message pool through the large language model and the corresponding answer information is generated based on the question information. Since the answer information is generated based on the user's question information and the real-time analysis results, it has high scalability and meets the diverse needs of users.

[0074] The passenger flow analysis method of the multimodal large model in the present application is applied to the passenger flow analysis system. The passenger flow analysis system includes a server and a client. The server is deployed with: streaming media service, a pre-trained multimodal large model, a message transmission service and a large language model.

[0075] Among them, streaming media service refers to a service that compresses multiple sequential media data and sends them in segments. The service does not need to download the complete video data file before use. When the streaming media service transmits data, the streaming media transmission protocol used may be, for example, RTMP / RTSP protocol, HTTP-FLV protocol and HLS protocol. Among them, the RTMP / RTSP protocol can be based on the transmission control transmission control protocol (TCP) for long connection, does not need to establish connections multiple times, and has low latency, usually only 1 second to 3 seconds. Both the HTTP-FLV protocol and the HLS protocol can complete the transmission of video data files, but their confidentiality is low. Users can choose streaming media protocols according to actual application scenarios to build streaming services. Among them, the streaming services adapted to the RTMP / RTSP protocol mainly include Nginx-rtmp-module and Nginx-http-flv-module, wherein Nginx-http-flv-module is a streaming media server based on Nginx-rtmp-module. The multimodal large model is a deep learning model that can simultaneously process and analyze different modal information, including text data, image data, and audio data. The large language model can understand and analyze natural language and output the analysis results in the form of natural language. In this embodiment, the large language model can be used in an interface chain to call multiple analysis algorithms to analyze natural language and output the results.

[0076] Next, refer to Figure 1 The process of the passenger flow analysis method based on the multimodal large model in this application is introduced. Figure 1 It is a flow chart of a passenger flow analysis method based on a multimodal large model provided in an embodiment of the present application.

[0077] S101. Obtain video data of a target area to be analyzed in real time through a streaming media service.

[0078] Optionally, the target area to be analyzed may be an area where stampede accidents may occur in public places such as transportation tracks, scenic spots and shopping malls, as well as an area where reverse traffic is restricted, an area where the direction of traffic is restricted such as an escalator area, etc.

[0079] Optionally, video data of the target area can be acquired through an image acquisition device, which can be a high-definition camera or the like.

[0080] S102. Perform real-time analysis on the video data through a multimodal large model to generate real-time analysis results, where the real-time analysis results include: object position results, queuing detection results, retrograde detection results, and fall detection results.

[0081] The object position result is used to indicate the location of each person in the target area. As an optional implementation, the object position result can be displayed in a two-dimensional drawing of the scene.

[0082] The queue detection result is used to indicate whether a queue occurs in each preset queue area in the target area. As an optional implementation, the queue detection result can be determined by whether the area where the queue personnel are located is larger than the preset queue minimum area.

[0083] The reverse detection result is used to indicate whether the forward direction of the person in the preset area is opposite to the specified forward direction of the preset area. The preset area may be an area with a specified forward direction, such as a preset escalator area. For example, the reverse detection result may indicate the presence of a descending person in an ascending escalator in the target area.

[0084] The fall detection result is used to indicate whether the person’s current posture is in a fall state.

[0085] Optionally, the multimodal large model can process data of multiple modes, including video data, image data, and text data. In this embodiment, the video data obtained by the streaming media service is input into the multimodal large model, and the preset conditions are input into the multimodal large model to generate real-time analysis results. Among them, the preset conditions can be image data and text data. Exemplarily, the image data can be a two-dimensional drawing of the scene and the escalator area in the drawing, etc., and the text data includes a queuing threshold, etc.

[0086] S103: Send the real-time analysis result to the message pool of the server through the message transmission service.

[0087] Optionally, the message transmission service can be a Kafka service. Specifically, Kafka is a distributed, high-throughput and highly scalable message queue system. The Kafka service can be applied to log collection systems and message systems to provide information collection and transmission. Specifically, Kafka can be used as a message middleware to achieve system decoupling, redundant storage, traffic peak shaving, buffering, asynchronous communication, scalability and recoverability, and can also persist information to disk to reduce the risk of data loss.

[0088] Optionally, the message pool of the server is a data structure in the form of a stack, and its data is stored in a linked list manner.

[0089] After the implementation analysis result is sent to the message pool of the server, as an optional implementation, the real-time analysis result can be directly sent from the message pool to the client through the message transmission service for real-time situation display. As another optional implementation, the real-time analysis result can be analyzed again using a large language model in the form of an interface chain, so as to output the answer information required by the customer. The specific steps are described in S104.

[0090] S104. Read the real-time analysis results from the message pool through the large language model, and generate answer information corresponding to the user question information according to the user question information sent by the client, the preset interface information, the preset prompt information and the real-time analysis results, and send the answer information to the client for display through the message transmission service. The interface information includes the parameter information of the analysis algorithm to be called by the large language model, and the prompt information includes the data format of the analysis algorithm.

[0091] Optionally, the client can provide the user with an interactive interface, which may include a display of real-time analysis results, and may also provide the user with a query input bar, where the user can input the question to be queried into the query input bar using natural language, and the client converts the natural language input by the user into user question information and inputs it into the server.

[0092] As an optional implementation, interface information and prompt information can be pre-set. The interface information includes a set of parameter information of each analysis algorithm to be called by the large language model, wherein the parameter information may be interface parameter information and return value information, etc. The prompt information includes the data format of the input and output of the analysis algorithm, and may also describe the interface function.

[0093] Among them, different analysis algorithms can have different interface parameter information, so that the large language model can be called through different interfaces to achieve analysis of specific content.

[0094] Optionally, the user question information, interface information, prompt information and real-time analysis results are input into the pre-trained large language model to generate answer information corresponding to the user question information. For example, the user question information is whether there is a person falling down, and the answer information generated by the large language model can be: there is a person falling down, and the coordinates of the person are at exit A on the second floor.

[0095] In this embodiment, first, video data is obtained in real time through the streaming media service, and the multimodal large model is used to perform real-time analysis on the video data to generate real-time analysis results. The real-time performance of the analysis result output can be improved by real-time analysis of the video data, so that the user can make corresponding actions on the personnel status of the current target area in a timely manner according to the real-time analysis results. In addition, the present embodiment can analyze the current passenger flow from multiple angles by analyzing the video data, with high scalability and high accuracy. Next, the real-time analysis results are sent to the message pool through the message transmission service. Since the message transmission service is timely, the real-time analysis results can be stored in the message pool in time to be called for display or further analysis. The further analysis process is to read the real-time analysis results from the message pool through the large language model, and generate the answer information corresponding to the user's question information according to the user's question information, interface information, prompt information and real-time analysis results. The real-time analysis results are further analyzed by the large language model, so as to provide the user with analysis results and suggestions to meet the diverse actual needs of users.

[0096] As an optional implementation, after the real-time analysis result is sent to the message pool of the server through the message transmission service, the real-time analysis result can also be sent to the client through the message transmission service for display.

[0097] Optionally, the message transmission service may be a Kafka service.

[0098] Optionally, the real-time analysis results are directly sent to the client and displayed to the user through an interactive interface. The interactive interface may include object position result display, queue detection result display, retrograde detection result display, and fall detection result display. It is worth mentioning that the result display in the above interactive interface is only an example, and the user can adjust the interactive interface display content according to actual needs.

[0099] Figure 2 This is a flow chart of a method for generating real-time analysis results provided by an embodiment of the present application. Figure 2 The step of performing real-time analysis on the video data and generating real-time analysis results in the above step S102 is introduced in detail.

[0100] S201, identifying objects in video data, determining initial position results, and converting the initial position results into object position results, where the initial position results are results under acquisition coordinates, and the object position results are coordinates under a two-dimensional drawing of the scene.

[0101] In this embodiment, the passenger flow is analyzed, so the object in the video data may be a person. However, this method is not limited to identifying people, but may also identify animals and objects, etc. This is not limited here.

[0102] Optionally, since the video data may be collected by a video acquisition device such as a camera, the initial position result obtained by object recognition in the video data has a perspective characteristic. For the position under the acquisition coordinates, it is necessary to convert the perspective image into coordinates under the scene two-dimensional drawing for intuitive analysis and display. The scene two-dimensional drawing may be a computer-aided design (CAD) plan view of the target area.

[0103] S202: Count the number of people in a preset queuing area in the target area according to the video data, and generate a queuing detection result according to the statistical result and a queuing threshold.

[0104] Specifically, firstly, the personnel in the preset queuing area in the target area are identified according to the video data, and then the personnel are counted to obtain the statistical result. It is worth mentioning that if the personnel is on the boundary line between the preset queuing area and the non-preset queuing area, the personnel will still be counted in the statistical result.

[0105] As an optional implementation, if the statistical result is greater than the queue threshold, the queue detection result is that a queue occurs. If the statistical result is not greater than the queue threshold, the queue detection result is that no queue occurs.

[0106] S203: judging the orientation, travel direction and preset escalator direction of people in the preset escalator area in the target area according to the video data, and generating a reverse detection result according to the judgment result.

[0107] Specifically, if it is detected that the direction of the person in the preset escalator area is the same as the travel direction, it is determined whether the travel direction is the same as the preset escalator direction. If so, the person has not shown any retrograde behavior. If the travel direction is different from the preset escalator direction, the person has shown any retrograde phenomenon. If it is detected that the direction of the person in the preset escalator area is different from the travel direction, it is determined whether the travel direction is the same as the preset escalator direction. If so, the person has shown any retrograde behavior. If the travel direction is different from the preset escalator direction, the person has not shown any retrograde phenomenon.

[0108] S204: Identify the posture of the person in the video data, and generate a fall detection result based on the identification result.

[0109] Specifically, the multimodal model can identify the posture of a person, including standing, squatting and falling. If the posture is detected as falling, the position of the fallen person is generated.

[0110] In this embodiment, by respectively determining the object position result, the queuing detection result, the reverse detection result and the fall detection result, the target result is detected and analyzed from multiple angles to meet various user needs.

[0111] Figure 3 is a flow chart of a method for converting position results provided in an embodiment of the present application. Figure 3 In the above process, the specific process of converting the initial position result into the object position result in step S201 is as follows:

[0112] S301, obtaining initial positions corresponding to a plurality of preset position points in an initial position result.

[0113] When the target area is preset and the video of the target area is captured by the video capture device, the position of the preset position point in the video data captured by the video capture device is determined. For example, if the shape of the target area is a rectangle, the four vertices of the rectangle are used as the preset position points, and the positions of the four fixed points in the video data captured by the video capture device are obtained.

[0114] S302: Obtain the target position corresponding to each preset position point.

[0115] Among them, the target position corresponding to each preset position point is the position of the preset position point in the two-dimensional coordinates of the scene.

[0116] For example, Figure 4 It is a schematic diagram of the relationship between an initial position and a target position of a preset position point provided in an embodiment of the present application. Figure 4 The middle left picture shows the initial positions of preset position point A, preset position point B, preset position point C and preset position point D when the shape of the target area is a rectangle, and the right picture shows the target positions of preset position point a, preset position point b, preset position point c and preset position point d when the shape of the target area is a rectangle.

[0117] S303: Determine coordinate conversion information according to the initial position corresponding to each preset position point and the target position corresponding to each preset position point.

[0118] Optionally, the coordinate conversion information may indicate a correspondence between an initial position corresponding to each preset position point and a target position corresponding to each preset position point.

[0119] S304: transform the initial position result according to the coordinate transformation information to obtain the object position result.

[0120] As an optional implementation, the coordinate transformation information may be the following formula (1):

[0121]

[0122] Among them, x, y and z together represent the coordinates of the target position, u and v are the coordinates of the initial position, is the correspondence between the initial position and the target position in the coordinate transformation information.

[0123] By transforming the above formula (1), we can obtain formulas (2), (3) and (4):

[0124] x=a 11 u+a 12 v+a 13 (2)

[0125] y=a 21 u+a 22 v+a 23 (3)

[0126] z=a 31 u+a 32 v+a 33 (4)

[0127] According to the above formulas (2), (3) and (4), the coordinates of the initial position are transformed, such as the following formulas (5) and (6):

[0128]

[0129]

[0130] Among them, x ′ and ′ is the coordinate of the target location.

[0131] In this embodiment, coordinate conversion information is determined based on the initial position corresponding to each preset position point and the target position corresponding to each preset position point, and then the object position result is determined based on the coordinate conversion information, so that the user can intuitively see the position of each person in the target area.

[0132] As an optional implementation, Figure 5 Schematic diagram of a training process of a multi-modal large model provided in an embodiment of the present application. Figure 5 As shown in the figure, the training process of the multimodal large model is as follows:

[0133] S501, inputting a first image sample into an initial image model, and inputting a first text sample into an initial text model, performing comparative training on the initial image model and the initial text model, and obtaining a target image model, a target text model, and a weight of the target text model.

[0134] The image and the text in the first image sample and the first text sample are associated with each other. Each sample can be pre-processed, for example, by adding noise.

[0135] The first image sample is input into the initial image model, and the first text sample is input into the initial text model. The initial image model and the initial text model are compared and learned to minimize the cross entropy loss of the similarity score, and the target image model, the target text model and the weight of the target text model are obtained.

[0136] As an optional implementation, the initial image model may be a Visual Transformer (VIT) model, and the initial text model may be a Language Learning Automaton Model (LLaMA).

[0137] S502: Establish an initial fine-tuning model based on the weights of the target text model, and input the first training sample into the initial fine-tuning model for training to obtain a target fine-tuning model.

[0138] Specifically, the initial fine-tuned model can inherit the weights of the target text model.

[0139] Optionally, the first training sample may include image data and text data, etc. As an optional implementation, the first training sample may be high-quality sample data, such as data with low noise.

[0140] Optionally, when the first training sample is input into the initial fine-tuning model for training, only the query layer and the cross-attention layer of the initial fine-tuning model may be trained.

[0141] As an optional implementation, the initial fine-tuning model may be a quantum language model (Quantum Language Model with Attention, QLLaMA).

[0142] S503. Input the second training sample into the target image model, the target text model and the target fine-tuning model respectively, and output the first prediction result, the second prediction result and the third prediction result respectively. The second training sample includes a second image sample, a second text sample and a prompt word, and input the first prediction result, the second prediction result and the third prediction result into the decoder to obtain a fourth prediction result. According to the first prediction result, the second prediction result, the third prediction result and the fourth prediction result, iteratively correct the target image model, the target text model, the target fine-tuning model and the decoder to obtain a multimodal large model.

[0143] The second training sample may also be high-quality sample data, such as data with low noise. The second image sample, the second text sample and the prompt word in the second training sample are respectively input into the target image model, the target text model and the target fine-tuning model, and the first prediction result, the second prediction result and the third prediction result are output respectively. The prompt word may be content input by the user to help the model understand the user's needs, ensuring that the multimodal large model can output more in line with the user's expectations. The prompt word can establish a connection between the data of each modality for the large model when the multimodal data is input into the multimodal large model.

[0144] Optionally, the decoder is used to generate the final output from the intermediate representation or hidden state of the multimodal large model, and can output results in multiple modalities according to user needs.

[0145] Optionally, the multimodal model includes an iteratively modified target image model, a target text model, a target fine-tuning model, and a decoder.

[0146] In this embodiment, by performing comparative training on the initial image model and the initial text model, the first training sample is input into the initial fine-tuning model for training to obtain a target fine-tuning model, the second training sample is input into the target image model, the target text model and the target fine-tuning model respectively, and the first prediction result, the second prediction result and the third prediction result are output respectively, the above results are input into the decoder to obtain a fourth prediction result, and the target image model, the target text model, the target fine-tuning model and the decoder are iteratively corrected according to the first prediction result, the second prediction result, the third prediction result and the fourth prediction result to obtain a multimodal large model. In this embodiment, the accuracy of the multimodal large model is improved by training and fine-tuning the image model and the text model, so that the output effect is stable.

[0147] Furthermore, in the above step S501, the initial image model and the initial text model are subjected to comparative training to obtain the target image model and the target text model. The specific steps are introduced as follows:

[0148] Optionally, a cross entropy loss calculation is performed based on an actual result of the image model output by the initial image model and an actual result of the text model output by the initial text model to obtain loss information.

[0149] Optionally, the initial image model and the initial text model are iteratively modified according to the loss information to obtain a target image model and a target text model.

[0150] Among them, the loss information is the cross entropy loss value.

[0151] Optionally, the cross entropy loss function of the actual result of the text model output by the initial text model can be as follows:

[0152]

[0153] Where L is the loss value, N is the total number of samples, d is the Euclidean distance of the sample features, and the Euclidean distance is obtained by the actual results of the image model and the actual results of the text model. y is a flag indicating whether the corresponding samples in the first image sample and the first text sample match, and y is usually preset. When y is 1, the two samples are similar or matched, and when y is 0, it means that they do not match, and margin is a preset threshold. When d is greater than or equal to margin, it means that the model correction is completed. The target image model and the target text model are obtained.

[0154] It should be understood that when y is 1, the loss value is ∑yd 2 , that is, if the Euclidean distance of originally similar samples in the feature space is large, it means that the current model needs to be corrected. When y is 0, the loss function is ∑(1-y)max(margin,0) 2 , that is, when two samples are dissimilar, the Euclidean distance in the feature space is small and the loss value becomes larger.

[0155] In this embodiment, the cross entropy loss is calculated based on the actual results of the image model output by the initial image model and the actual results of the text model output by the initial text model to obtain loss information, and the initial image model and the initial text model are iteratively corrected based on the loss information to obtain the target image model and the target text model, thereby obtaining a large multimodal model to process multimodal data.

[0156] Figure 6 1 is a flow chart of a method for generating answer information proposed in an embodiment of the present application. Figure 6 As shown, the specific steps of generating answer information corresponding to the user question information according to the user question information sent by the client, the preset interface information, the preset prompt information and the real-time analysis result in the above step S104 are introduced below:

[0157] S601: Select a target interface from multiple interfaces based on user question information, interface information and prompt information.

[0158] Specifically, the content to be analyzed and the analysis algorithm are determined based on the user question information, interface information and prompt information, and then the target interface is determined based on the content to be analyzed and the analysis algorithm.

[0159] It is important to note that each analysis algorithm has a unique interface.

[0160] S602: calling a target interface, and executing an analysis algorithm by the target interface to obtain an intermediate result according to the real-time analysis result.

[0161] Specifically, the same user question information can call multiple analysis algorithms and obtain multiple intermediate results.

[0162] S603: Extract answer information based on the user's question information, intermediate results, and real-time analysis results.

[0163] Optionally, answer information may be extracted based on a large language model, according to user question information, intermediate results, and real-time analysis results.

[0164] In this embodiment, the intermediate results are obtained by determining the target interface, and the answer information is extracted according to the user's question information, the intermediate results and the real-time analysis results, thereby meeting the diverse customer flow analysis needs of users.

[0165] The present application provides a passenger flow analysis system, which includes a server and a client. The server is deployed with: a streaming media service, a pre-trained multimodal large model, a message transmission service, and a large language model:

[0166] The server is used to obtain video data of the target area to be analyzed in real time through the streaming media service;

[0167] The server is used to perform real-time analysis on the video data through the multimodal large model to generate real-time analysis results, wherein the real-time analysis results include: object position results, queuing detection results, retrograde detection results, and fall detection results;

[0168] The server is used to send the real-time analysis result to the message pool of the server through the message transmission service;

[0169] The server is used to read the real-time analysis result from the message pool through the large language model, and generate answer information corresponding to the user question information according to the user question information sent by the client, preset interface information, preset prompt information and the real-time analysis result, and send the answer information to the client for display through the message transmission service, the interface information includes parameter information of the analysis algorithm to be called by the large language model, and the prompt information includes the data format of the analysis algorithm.

[0170] Optionally, the server is used to identify the object in the video data, determine an initial position result, and convert the initial position result into the object position result, wherein the initial position result is a result under acquisition coordinates, and the object position result is a coordinate under a two-dimensional drawing of the scene;

[0171] According to the video data, the number of people in the preset queuing area in the target area is counted, and the queuing detection result is generated according to the statistical result and the queuing threshold;

[0172] According to the video data, the orientation, travel direction and preset escalator direction of the personnel in the preset escalator area in the target area are judged, and the reverse detection result is generated according to the judgment result;

[0173] The posture of the person in the video data is identified, and the fall detection result is generated according to the identification result.

[0174] Optionally, the server is used to obtain initial positions corresponding to a plurality of preset position points in the initial position result;

[0175] Obtaining the target position corresponding to each of the preset position points;

[0176] Determining coordinate conversion information according to the initial position corresponding to each of the preset position points and the target position corresponding to each of the preset position points;

[0177] The initial position result is transformed according to the coordinate transformation information to obtain the object position result.

[0178] Optionally, the server is used to input the first image sample into an initial image model, and input the first text sample into an initial text model, perform comparative training on the initial image model and the initial text model, and obtain a target image model, a target text model, and a weight of the target text model;

[0179] Establishing an initial fine-tuning model based on the weights of the target text model, and inputting the first training sample into the initial fine-tuning model for training to obtain a target fine-tuning model;

[0180] The second training sample is respectively input into the target image model, the target text model and the target fine-tuning model, and the first prediction result, the second prediction result and the third prediction result are respectively output, wherein the second training sample includes a second image sample, a second text sample and a prompt word, and the first prediction result, the second prediction result and the third prediction result are input into the decoder to obtain a fourth prediction result, and according to the first prediction result, the second prediction result, the third prediction result and the fourth prediction result, the target image model, the target text model, the target fine-tuning model and the decoder are iteratively corrected to obtain the multimodal large model.

[0181] Optionally, the server is used to perform cross entropy loss calculation according to an actual result of the image model output by the initial image model and an actual result of the text model output by the initial text model to obtain loss information;

[0182] The initial image model and the initial text model are iteratively modified according to the loss information to obtain a target image model and the target text model.

[0183] Optionally, the server is used to select a target interface from a plurality of interfaces based on the user question information, the interface information and the prompt information;

[0184] Calling the target interface, and executing the analysis algorithm by the target interface to obtain an intermediate result according to the real-time analysis result;

[0185] The answer information is extracted based on the user question information, the intermediate results and the real-time analysis results.

[0186] Optionally, the server is used to send the real-time analysis result to the client for display via the message transmission service.

[0187] The present application also provides a server, such as Figure 7 As shown, it is a structural diagram of a server provided in an embodiment of the present application, including: a processor 701, a memory 702 and a bus. The memory 702 stores machine-readable instructions executable by the processor 701. When the server is running, the processor 701 communicates with the memory 702 through the bus. When the machine-readable instructions are executed by the processor 701, the processing of the passenger flow analysis method based on the multimodal large model is performed.

[0188] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned passenger flow analysis method based on the multimodal large model are executed.

[0189] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0190] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0191] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A passenger flow analysis method based on a multimodal large model, characterized in that: Applied to a passenger flow analysis system, the passenger flow analysis system includes a server and a client, the server is deployed with: a streaming media service, a pre-trained multimodal large model, a message transmission service and a large language model, the method includes: Acquire video data of the target area to be analyzed in real time through the streaming media service; Performing real-time analysis on the video data through the multimodal large model to generate real-time analysis results, wherein the real-time analysis results include: object position results, queuing detection results, retrograde detection results, and fall detection results; Sending the real-time analysis result to the message pool of the server through the message transmission service; The real-time analysis result is read from the message pool through the large language model, and answer information corresponding to the user question information is generated according to the user question information sent by the client, the preset interface information, the preset prompt information and the real-time analysis result, and the answer information is sent to the client for display through the message transmission service, the interface information includes parameter information of the analysis algorithm to be called by the large language model, and the prompt information includes the data format of the analysis algorithm.

2. The passenger flow analysis method based on a multimodal large model according to claim 1 is characterized in that: Performing real-time analysis on the video data to generate real-time analysis results, including: Identify the object in the video data, determine the initial position result, and convert the initial position result into the object position result, wherein the initial position result is the result under the acquisition coordinates, and the object position result is the coordinate under the two-dimensional drawing of the scene; According to the video data, the number of people in the preset queuing area in the target area is counted, and the queuing detection result is generated according to the statistical result and the queuing threshold; According to the video data, the orientation, travel direction and preset escalator direction of the personnel in the preset escalator area in the target area are judged, and the reverse detection result is generated according to the judgment result; The posture of the person in the video data is identified, and the fall detection result is generated according to the identification result.

3. The passenger flow analysis method based on multimodal large model according to claim 2 is characterized in that: The converting the initial position result into the object position result comprises: Obtaining initial positions corresponding to a plurality of preset position points in the initial position result; Obtaining the target position corresponding to each of the preset position points; Determining coordinate conversion information according to the initial position corresponding to each of the preset position points and the target position corresponding to each of the preset position points; The initial position result is transformed according to the coordinate transformation information to obtain the object position result.

4. The passenger flow analysis method based on a multimodal large model according to claim 1 is characterized in that: The training process of the multimodal large model includes: Inputting a first image sample into an initial image model and a first text sample into an initial text model, performing comparative training on the initial image model and the initial text model, and obtaining a target image model, a target text model, and a weight of the target text model; Establishing an initial fine-tuning model based on the weights of the target text model, and inputting the first training sample into the initial fine-tuning model for training to obtain a target fine-tuning model; The second training sample is respectively input into the target image model, the target text model and the target fine-tuning model, and the first prediction result, the second prediction result and the third prediction result are respectively output, the second training sample includes a second image sample, a second text sample and a prompt word, and the first prediction result, the second prediction result and the third prediction result are input into the decoder to obtain a fourth prediction result, and the target image model, the target text model, the target fine-tuning model and the decoder are iteratively corrected according to the first prediction result, the second prediction result, the third prediction result and the fourth prediction result to obtain the multimodal large model.

5. The passenger flow analysis method based on multimodal large model according to claim 4 is characterized in that: The comparative training of the initial image model and the initial text model to obtain the target image model and the target text model includes: Performing cross entropy loss calculation according to an actual result of the image model output by the initial image model and an actual result of the text model output by the initial text model to obtain loss information; The initial image model and the initial text model are iteratively modified according to the loss information to obtain a target image model and the target text model.

6. The passenger flow analysis method based on multimodal large model according to claim 1 is characterized in that: The generating of answer information corresponding to the user question information according to the user question information sent by the client, the preset interface information, the preset prompt information and the real-time analysis result includes: Selecting a target interface from among multiple interfaces based on the user question information, the interface information, and the prompt information; Calling the target interface, and executing the analysis algorithm by the target interface to obtain an intermediate result according to the real-time analysis result; The answer information is extracted based on the user question information, the intermediate results and the real-time analysis results.

7. The passenger flow analysis method based on multimodal large model according to claim 1 is characterized in that: The method further comprises: The real-time analysis result is sent to the client for display via the message transmission service.

8. A passenger flow analysis system, characterized in that: The passenger flow analysis system includes a server and a client. The server is deployed with: a streaming media service, a pre-trained multimodal large model, a message transmission service, and a large language model: The server is used to obtain video data of the target area to be analyzed in real time through the streaming media service; The server is used to perform real-time analysis on the video data through the multimodal large model to generate real-time analysis results, wherein the real-time analysis results include: object position results, queuing detection results, retrograde detection results, and fall detection results; The server is used to send the real-time analysis result to the message pool of the server through the message transmission service; The server is used to read the real-time analysis result from the message pool through the large language model, and generate answer information corresponding to the user question information according to the user question information sent by the client, preset interface information, preset prompt information and the real-time analysis result, and send the answer information to the client for display through the message transmission service, the interface information includes parameter information of the analysis algorithm to be called by the large language model, and the prompt information includes the data format of the analysis algorithm.

9. A server, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the server is running, the processor executes the machine-readable instructions to perform the steps of the passenger flow analysis method based on a multimodal large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the passenger flow analysis method based on a multimodal large model as described in any one of claims 1 to 7.

Images