A method and system for automatically generating a light show script based on AIGC semantic understanding

Abstract

The application discloses a light show script automatic generation method and system based on AIGC semantic understanding, and belongs to the technical field of light control, wherein the system comprises a data acquisition unit, a man-machine interaction unit, a light show script generation unit, an AIGC multi-modal content generation unit and a control instruction generation unit; the data acquisition unit is in communication connection with the light show script generation unit and the control instruction generation unit respectively; the light show script generation unit is in communication connection with the man-machine interaction unit and the AIGC multi-modal content generation unit respectively; and the AIGC multi-modal content generation unit is in communication connection with the control instruction generation unit. The application uses physical device capability as a constraint condition for light show script generation, and organically combines the physical device capability with user creativity, thereby avoiding the situation that virtual effects of the light show script are disconnected with physical devices.

Description

Technical Field

[0001] This invention relates to the field of lighting control technology. Specifically, it relates to a method and system for automatically generating light show scripts based on AIGC semantic understanding. Background Technology

[0002] A light show is a form of public art that uses light and shadow technology as its core, integrating elements such as architecture and music. It is often used to showcase urban culture and create a festive atmosphere. Light shows are usually accompanied by music, videos, and other expressive elements, requiring seamless coordination between light changes and these elements. With the use of artificial intelligence (AI), AI technology is increasingly being used to generate light show scripts. However, the use of AI still requires users to provide constraints and frameworks to ensure that the AI ​​output better meets the user's pre-set needs. How to enable AI to play a greater role in the light show script generation process while avoiding a disconnect between virtual effects and physical equipment is a key issue that needs to be considered when using AI to generate light show scripts.

[0003] Therefore, the technical problem to be solved by the present invention is to provide a method and system for automatically generating light show scripts based on AIGC semantic understanding. By using physical device capabilities as constraints for generating light show scripts, the physical device capabilities are organically combined with user creativity, thus avoiding the situation where the virtual effects of the light show scripts are disconnected from the physical devices.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0005] A method for automatically generating light show scripts based on AIGC semantic understanding includes the following steps:

[0006] (S10) Collect spatial metadata, equipment metadata, semantic capability metadata, environmental metadata and material basic data. Among them, spatial metadata includes the on-site three-dimensional dimensions of the light show, coordinates inside the building, landscape nodes and installation coordinates of the lighting equipment; equipment metadata includes the lighting equipment model, color gamut, brightness range, response speed and control protocol; semantic capability metadata is the light effects that the equipment can present; environmental metadata includes coordinate system, viewpoint, perspective relationship and camera intrinsic / extrinsic parameters; material basic data includes pre-stored videos, pictures and lighting effect scripts.

[0007] (S20) Collect users' natural language creative ideas;

[0008] (S30) The parsing module of the fine-tuned LLM model is used to perform intent recognition, keyword extraction, style classification and sentiment quantification on the user natural language creative collected in step S20, and output a structured demand feature vector V=[S,C,B,T,D], where S is the application scenario, C is the style vector, B is the rhythm quantification, T is the total duration and D is the device and environment information.

[0009] (S40) Integrate the demand feature vector V generated in step S30 with the device metadata and semantic metadata collected in step S10 into a complete input context and input it into the light show script generation module of the fine-tuned LLM large model to output the light show script.

[0010] (S50) Perform device hardware verification on the light show script output in step S40, replace the light and shadow effects and color parameters that are not supported by the existing devices in the light show script with equivalent light effects and color parameters of the same style and that can be achieved by the existing devices, and output the corrected light show script.

[0011] (S60) Using AIGC technology, the corrected light show script output in step S50 is used to generate a corresponding video segment sequence and generate or match the corresponding music, and outputs a video sequence corresponding to the light show script, an independent synchronous audio track aligned with the video sequence at the millisecond level, and a time-emotion rhythm curve generated based on the independent synchronous audio track.

[0012] (S70) Using the video sequence generated in step S60, the time-emotion rhythm curve, and the digital twin of the light show venue constructed using the data collected in step S10, light equipment lighting effect control instructions are generated.

[0013] (S80) The independent synchronized audio track generated in step S60, the lighting equipment control instructions generated in step S70, and the timeline and Cue point information determined by the light show script generated in step S40 are aligned and encapsulated on the same timeline with millisecond-level precision to generate a standardized light show performance file that can be played offline and supports direct loading by the on-site controller.

[0014] The above-mentioned AIGC semantic understanding-based automatic light show script generation method, in step S40, when the fine-tuned LLM model outputs the light show script, restricts the format of the light show script output by the fine-tuned LLM model according to the strongly constrained standardized prompt words, and generates a light show script with a fixed four-segment narrative structure according to the following constraint rules:

[0015] (R1) The narrative structure is forcibly divided into four sections: opening, development, climax, and ending;

[0016] (R2) Mark each script segment with the time range, applicable lighting area, main lighting effect type, dominant color, movement rhythm, and accompanying audio mood;

[0017] (R3) All light effect description words used in the light show script can only be selected from the semantic capability metadata collected in step S10.

[0018] The above-mentioned AIGC-based automatic script generation method for light shows, in step S60, before outputting the video sequence, uses a multiphysics decomposition total loss function to constrain and enhance the video sequence; wherein, the multiphysics decomposition total loss function is:

[0019] L total =L temporal +λ p (t)×L multi-physics +λ l ×L lighting +λ c ×L color ;

[0020] Among them, L temporal The temporal continuity loss is calculated using the following formula:

[0021] ;

[0022] In the formula, I t and I t-1 t and (t-1)th frames of video, respectively; H is the height of the video frame; W is the width of the video frame; ‖*‖ is the L2 norm, which characterizes the intensity of pixel differences between frames;

[0023] L lighting The loss of lighting consistency is calculated using the following formula:

[0024] ;

[0025] In the formula, The estimated light source direction for the t-th frame of the video; The estimated light source direction for the (t-1)th frame of the video; The intensity estimated for the t-th frame of the video; The intensity estimated for the (t-1)th frame of the video;

[0026] L color The loss of color consistency is calculated using the following formula:

[0027] ;

[0028] In the formula, N is the number of sampled pixels, and its value is N=(H×W) / 64; p i For sampling pixels; flow(p)i ) represents the video from frame t-1 to frame t in p i Optical flow at a point;

[0029] λ p (t) represents the dynamic weights of the physical constraints, which are calculated using the following formula:

[0030] λ p (t)=σ(a·v avg (t)+b·v avr (t)+c);

[0031] In the formula, v avg (t) represents the average velocity of the optical flow field in the t-th frame of the video; v avr (t) represents the variance of the optical flow field in the t-th frame of the video; a = 2.0, b = 1.0, c = -1.0; σ is the Sigmoid function;

[0032] λ l λ represents the weight for light and shadow consistency. c This represents the color consistency weight, and is related to λ. p (t) and λ l Satisfying λ l =λ c =0.5×(1-λ p (t));

[0033] L multi-physics The multiphysics loss is calculated using the following formula:

[0034] L multi-physics =w g ×L gravity +w i ×L inertia +w f ×L fliud +w e ×L elastic ;

[0035] In the formula, L gravity For gravitational field constraints, the following formula is used for calculation:

[0036] ;

[0037] Among them, a t is the motion acceleration estimated for the optical flow field; g is the standard gravitational acceleration vector;

[0038] L inertia For inertial motion constraints, the following formula is used for calculation:

[0039] ;

[0040] Among them, vt v is the average motion velocity of the optical flow in the t-th frame of the video; t-1 The average motion velocity of the optical flow in the (t-1)th frame of the video;

[0041] L fliud Assuming fluid dynamic constraints, the following formula is used for calculation:

[0042] ;

[0043] in, To perform Laplacian operations on intensity or grayscale images of video frames;

[0044] L elastic For elastic deformation constraints, the following formula is used for calculation:

[0045] ;

[0046] Where, d t The deformation displacement of the characteristic point;

[0047] w g w i w f and w e These are the gravitational field weight, inertial field weight, hydrodynamic weight, and elastic field weight, respectively.

[0048] The aforementioned method for automatically generating light show scripts based on AIGC semantic understanding, in step S70, utilizes a physical space-pixel space joint solution function to generate lighting equipment effect control instructions by applying projection mapping, capability constraints, response compensation, and rhythm adjustment to the color values ​​of video pixels in the video sequence; wherein, the physical space-pixel space joint solution function is:

[0049] Out(d,t)=M(I(u(d,t),v(d,t),t),C tech (d),τ(d),ρ(t));

[0050] In the formula, M(*) represents a multi-dimensional mapping;

[0051] I(u(d,t),v(d,t),t) represents the color value of the video at pixel coordinates u(d,t) and v(d,t) and time t.

[0052] (u(d,t),v(d,t))=Proj(P(d)) represents the pixel coordinates of the device d in the world coordinate system, which are mapped to the video frame by the projection mapping function Proj.

[0053] C tech (d) represents the technical capability parameters of equipment d;

[0054] τ(d) is the response time of device d;

[0055] ρ(t) is the real-time rhythm curve of the video, which is calculated using the rhythm quantization B obtained in step S30 through the following formula:

[0056] ;

[0057] Where P is the beat period, P=60 / B; A is the beat intensity coefficient, with a value range of [0.1,0.5]; E(t) is the base value of emotional intensity, calculated by the following formula:

[0058] ;

[0059] Among them, t k For the k-th critical time point, 0 = t0 < t1 < ... < t N-1 =T, where T is the total duration; e k For time t k The emotional intensity, with a value range of [0,1].

[0060] The above-mentioned AIGC-based automatic script generation method for light shows converts the installation coordinates of the lighting equipment into video pixel coordinates in Proj(P(d)), establishing a spatial correspondence between physical equipment and video content. Specifically, it is converted into:

[0061] ;

[0062] Where, P(d)=(X w ,Y w Z w ,1) T Let d be the homogeneous world 3D coordinates of device d;

[0063] K is the virtual camera intrinsic parameter matrix (3×3);

[0064] [R|t] is the virtual camera extrinsic parameter matrix (3×4), where R is the rotation matrix used to control the rotation attitude of the camera lens, and t is the translation vector used to control the three-dimensional position of the camera. It defines the position and attitude of the virtual camera in the homogeneous world coordinate system.

[0065] The aforementioned method for automatically generating light show scripts based on AIGC semantic understanding also includes:

[0066] (S90) Using the standardized light show performance file generated in step S80, the light show is simulated and previewed in a virtual scene constructed using the data collected in step S10 to achieve the final effect in a real environment;

[0067] (S100) Based on the final effect of step S90, the standardized light show performance file generated in step S80 is optimized to form the final version of the performance file and the lighting equipment control instruction set.

[0068] A system for generating light show scripts using the aforementioned AIGC-based semantic understanding-based automatic light show script generation method includes:

[0069] The data acquisition unit is used to collect spatial metadata, device metadata, semantic capability metadata, environmental metadata, and basic material data;

[0070] The human-computer interaction unit is used to input users' natural language ideas and natural language requirements.

[0071] The light show script generation unit is used to generate light show scripts based on natural language creative ideas and natural language requirements. The light show script generation unit has a built-in fine-tuned LLM model, which includes a parsing module and a light show script generation module. The parsing module is used to perform intent recognition, keyword extraction, style classification, and sentiment quantification on the user's natural language creative ideas and natural language requirements, and outputs a requirement feature vector V. The light show script generation module is used to generate a light show script using the data collected by the data acquisition unit and the requirement feature vector V output by the parsing module.

[0072] The AIGC multimodal content generation unit is used to generate an AIGC video sequence, an independent synchronous audio track aligned with the AIGC video sequence in milliseconds, and a time-emotion rhythm curve based on the light show script.

[0073] The control instruction generation unit is used to generate the final version of the performance file and lighting equipment control instruction set based on the AIGC video sequence, the independent synchronous audio track aligned to the AIGC video sequence in milliseconds, the time-emotion rhythm curve, and the data collected by the data acquisition unit;

[0074] The data acquisition unit is connected to the light show script generation unit and the control instruction generation unit, respectively. The light show script generation unit is connected to the human-computer interaction unit and the AIGC multimodal content generation unit, respectively. The AIGC multimodal content generation unit is connected to the control instruction generation unit.

[0075] The system also includes a constraint enhancement unit for performing multi-condition constraint processing on AIGC video sequences, an AIGC multimodal content generation unit that is communicatively connected to the constraint enhancement unit, and a constraint enhancement unit that is communicatively connected to the control instruction generation unit.

[0076] The system also includes a central control unit, which is communicatively connected to the data acquisition unit, the human-computer interaction unit, the light show script generation unit, the AIGC multimodal content generation unit, and the control command generation unit.

[0077] The technical solution of the present invention achieves the following beneficial technical effects:

[0078] This invention parses user natural language creative ideas into structured script requirements, and then combines the structured script requirements with physical equipment capabilities and related information, so that the virtual effects of the light show script generated according to the script requirements can be realized by physical equipment, thus avoiding the disconnect between the virtual effects of the light show and the physical equipment. Attached Figure Description

[0079] Figure 1 The working principle diagram of the AIGC-based automatic light show script generation system in this invention;

[0080] Figure 2 This is a flowchart illustrating the automatic generation of light show scripts based on AIGC semantic understanding in this invention. Detailed Implementation

[0081] like Figure 1 As shown, the AIGC-based automatic light show script generation system of this invention includes a central control unit, a data acquisition unit, a human-computer interaction unit, a light show script generation unit, an AIGC multimodal content generation unit, a control instruction generation unit, and a constraint enhancement unit. The central control unit is communicatively connected to the data acquisition unit, the human-computer interaction unit, the AIGC multimodal content generation unit, the constraint enhancement unit, and the control instruction generation unit. The data acquisition unit is communicatively connected to the light show script generation unit and the control instruction generation unit. The light show script generation unit is communicatively connected to the human-computer interaction unit and the AIGC multimodal content generation unit. The content generation unit communicates with the AIGC multimodal content generation unit, the constraint enhancement unit communicates with the control instruction generation unit, and the light show script generation unit communicates with the control instruction generation unit. The light show script generation unit has a built-in fine-tuned LLM model, which includes a parsing module and a light show script generation module. The parsing module is used to perform intent recognition, keyword extraction, style classification, and sentiment quantification on the user's natural language creativity and natural language needs, and outputs a demand feature vector V. The light show script generation module is used to generate a light show script using the data collected by the data acquisition unit and the demand feature vector V output by the parsing module.

[0082] In this invention, the automatic generation system for light show scripts based on AIGC semantic understanding will be described based on how to perform automatic generation of light show scripts based on AIGC semantic understanding.

[0083] Specifically, such as Figure 2 As shown, the automatic generation of light show scripts based on AIGC semantic understanding is achieved through the following steps:

[0084] (S10) Collect spatial metadata, device metadata, semantic capability metadata, environmental metadata and material basic data.

[0085] This step completes the full mapping from the physical world to the digital world, constructing the foundational data base for light show creation. It provides precise spatial, equipment, and capability constraint data for subsequent semantic analysis, script generation, light effect compositing, and equipment mapping, preventing a disconnect between virtual effects and physical equipment. Specifically, the data collected in this step includes:

[0086] S1.1 Spatial Metadata: Obtain the on-site 3D dimensions of the light show, building facade coordinates, landscape nodes, and light strip / light fixture installation coordinates (a set of light strip / light fixture installation positions (X / Y / Z)) for constructing a 1:1 digital twin site model;

[0087] S1.2 Device Metadata: Collect data on models, color gamut, brightness range, response speed, and control protocols related to full-color LED strips, smart LED lights, and projection equipment to establish a device capability library;

[0088] S1.3 Semantic Capability Metadata: Label each device with renderable light effect tags (flow, gradient, scan, flash, outline, fill);

[0089] S1.4 Environmental metadata: coordinate system, viewpoint, perspective relationship, camera intrinsic / extrinsic parameters;

[0090] S1.5 Material Basic Data: Pre-stores basic materials such as videos, images, and lighting effect scripts, and supports video encoding and decoding, and multi-layer fusion.

[0091] (S20) Collect users' natural language creative ideas.

[0092] This step aims to provide a constraint framework for generating the light show script, such as changes in light color and the shapes (lines, points, graphics, etc.) that the light shows represent. The constraint framework consists of user-defined limits and descriptions of the light effects. For example, if the user inputs an unstructured creative description: "A 5-minute light show featuring simple lines, monochrome gradients, and breathing light effects, emphasizing a sophisticated, restrained, and modern art style, suitable for high-end buildings, science and technology museums, cultural centers, and other similar settings."

[0093] (S30) The parsing module in the fine-tuned LLM model is used to perform intent recognition, keyword extraction, style classification and sentiment quantification on the user natural language creative collected in step S20, and output a structured demand feature vector V=[S,C,B,T,D], where S is the application scenario, C is the style vector, B is the rhythm quantification, T is the total duration and D is the device and environment information.

[0094] Among them, the application scenario is specifically such as high-end buildings, science and technology museums or / and cultural museums; the style vector is specifically such as simple lines 0.9, monochrome gradient 0.8, breathing light effect 0.7, and high-end or modern art style 0.5; the rhythm quantification is specifically the average BPM, used to form the emotional intensity curve; the total duration is specifically such as 300 seconds; and the equipment and environmental information can be extracted from the data collected in step S10.

[0095] (S40) The requirement feature vector V generated in step S30 is integrated with the device metadata and semantic metadata (including the light effect labels supported by each lamp, lamp installation area, color gamut, brightness, hardware capability constraints, etc.) collected in step S10 into a complete input context and input into the light show script generation module in the fine-tuned LLM large model to output the light show script.

[0096] In this step, when the fine-tuned LLM model outputs the light show script, the format of the light show script is restricted by the strongly constrained standardized prompts. A light show script with a fixed four-segment narrative structure is generated according to the following constraint rules:

[0097] (R1) The narrative structure is forcibly divided into four paragraphs: opening, development, climax and ending. The time interval of each paragraph is automatically allocated according to the total duration parameter T in the feature vector, and the change pattern of the built-in emotional intensity curve of the feature vector is matched.

[0098] (R2) Mark each script segment with the time range, applicable lighting area, main lighting effect type, dominant color, movement rhythm, and accompanying audio mood;

[0099] (R3) All light effect descriptions used in the light show script can only be selected from the semantic capability metadata collected in step S10, so as to prevent the appearance of light effect descriptions that cannot be achieved by the on-site lighting fixtures.

[0100] (S50) Perform device hardware verification on the light show script output in step S40, replace the light and shadow effects and color parameters that are not supported by the existing devices in the light show script with equivalent light effects and color parameters of the same style and that can be achieved by the existing devices, and output the corrected light show script.

[0101] After the fine-tuned LLM large model outputs the light show script, the AIGC-based automatic light show script generation system automatically executes this step. Specifically, it iterates through all descriptions related to light effects, color, and brightness in the light show script and compares them item by item with the device metadata capability library. If the script contains light and shadow effects that the current venue lighting fixtures do not support, or color parameters that exceed the color gamut / brightness range, the AIGC-based automatic light show script generation system automatically replaces them with equivalent light effects and color parameters of the same style that can be implemented by the hardware. After the correction is completed, a structured segmented light show script is output, which is the corrected light show script.

[0102] (S60) Using AIGC technology, generate the corresponding video segment sequence and generate or match the corresponding music according to the corrected light show script output in step S50, and output the video sequence corresponding to the light show script, the independent synchronized audio track aligned with the video sequence at the millisecond level, and the time-emotion rhythm curve.

[0103] In this step, the video sequence is generated by the Wensheng video model using visual keywords, dominant colors, and motion styles of each scene in the structured segmented light show script as conditional prompts. The content in the video sequence is precisely aligned with the script's timeline (millisecond level). Simultaneously, based on the emotional tone and music suggestions in the light show script, corresponding music is generated or matched through an audio AIGC model and deeply synchronized with the rhythm analysis results of the visual content.

[0104] To improve the temporal smoothness, physical compliance, lighting stability, and color consistency of video sequences, this step also enhances the constraints on the video sequences produced by the Wensheng video model through the multi-physics decomposition total loss function, upgrading the traditional single physical constraint into four combinatorial subfields: gravity, inertia, fluid, and elasticity.

[0105] Specifically, in order to uniformly optimize multi-dimensional quality and avoid overall imbalance caused by optimizing a single index, the multiphysics decomposition total loss function introduced in this step is:

[0106] L total =L temporal +λ p (t)×L multi-physics +λ l ×L lighting +λ c ×L color ;

[0107] Among them, L temporal To mitigate the loss of temporal continuity, ensure smooth and continuous lighting animation, and meet the low latency requirements of light shows, the following formula is used for calculation:

[0108] ;

[0109] In the formula, I t and I t-1 t and (t-1)th frames of video, respectively; H is the height of the video frame; W is the width of the video frame; ‖*‖ is the L2 norm, which characterizes the intensity of pixel differences between frames;

[0110] L lighting To minimize the loss of light and shadow consistency, achieve a sophisticated feel in architectural / landscape lighting, and eliminate cluttered light and shadow, the following formula is used for calculation:

[0111] ;

[0112] In the formula, The estimated light source direction for the t-th frame of the video; The estimated light source direction for the (t-1)th frame of the video; The intensity estimated for the t-th frame of the video; The intensity estimated for the (t-1)th frame of the video;

[0113] L color To minimize color consistency loss and prevent abrupt color changes in lighting, the following formula is used for calculation:

[0114] ;

[0115] In the formula, N is the number of sampled pixels, with a value of N=(H×W) / 64, balancing calculation accuracy and efficiency; p i For sampling pixels; flow(p) i ) represents the video from frame t-1 to frame t in p i Optical flow (motion vector) at a point;

[0116] λ p (t) represents the dynamic weight of physical constraints, which adaptively adjusts according to the scene type. It can be calculated based on optical flow statistical characteristics and obtained by the following formula:

[0117] λ p (t)=σ(a·v avg (t)+b·v avr (t)+c);

[0118] In the formula, v avg (t) represents the average velocity of the optical flow field in the t-th frame of the video; v avr (t) represents the variance of the optical flow field in the t-th frame of the video; a = 2.0, b = 1.0, c = -1.0; σ is the Sigmoid function;

[0119] λ l The lighting consistency weight is adjusted to control the intensity of lighting constraints, balancing overall brightness with local lighting details; λ cTo adjust the color consistency weight, the color stability constraint strength is adjusted to ensure natural color transitions, and it is consistent with λ. p (t) and λ l Satisfying λ l =λ c =0.5×(1-λ p (t));

[0120] L multi-physics The multiphysics loss is calculated using the following formula:

[0121] L multi-physics =w g ×L gravity +w i ×L inertia +w f ×L fliud +w e ×L elastic ;

[0122] In the formula, L gravity For gravitational field constraints, the following formula is used for calculation:

[0123] ;

[0124] Among them, a t is the motion acceleration estimated for the optical flow field; g is the standard gravitational acceleration vector, used to constrain the falling motion of light and shadow to conform to the laws of gravity;

[0125] L inertia For inertial motion constraints, the following formula is used for calculation:

[0126] ;

[0127] Among them, v t v is the average motion velocity of the optical flow in the t-th frame of the video, used to ensure smooth motion of the light effect without abrupt changes; t-1 The average motion velocity of the optical flow in the (t-1)th frame of the video;

[0128] L fliud For fluid dynamic constraints, used to smooth the diffusion of light waves and water flow effects, the following formula is used for calculation:

[0129] ;

[0130] in, To perform Laplacian operations on intensity or grayscale images of video frames;

[0131] L elastic As an elastic deformation constraint, it is used to constrain the stretching and rebound effect of the light strip to conform to the elastic law, and is calculated by the following formula:

[0132] ;

[0133] Where, d t The deformation displacement of the characteristic point;

[0134] w g w i w f and w e These are the weights for the gravitational field, inertial field, hydrodynamics, and elastic field. These four weights are dynamically allocated based on the scene type. Different physical fields are activated for different lighting themes. The dynamic allocation mechanism for the weights of each physical field loss based on the current scene's motion type is shown in the table below:

[0135] Table 1 w g w i w f and w e Weight values ​​in different scenarios

[0136]

[0137] (S70) Using the video sequence generated in step S60, the time-emotion rhythm curve, and the digital twin constructed using the data collected in step S10, a lighting control command for the lighting equipment (DMX512 / Art-Net protocol) is generated.

[0138] This step aims to achieve a precise, low-latency, and highly stable conversion from "video pixels" to "device color".

[0139] In this step, by introducing a physical space-pixel space joint solution function, the color values ​​of video pixels are converted into the final control commands of the device after projection mapping, capability constraints, response compensation, and rhythm modulation. The physical space-pixel space joint solution function is as follows:

[0140] Out(d,t)=M(I(u(d,t),v(d,t),t),C tech (d),τ(d),ρ(t));

[0141] In the formula, M(*) is a multi-dimensional mapping to ensure the visual consistency of "video color = lighting color";

[0142] I(u(d,t),v(d,t),t) represents the color value of the video at pixel coordinates u(d,t) and v(d,t) and time t.

[0143] (u(d,t),v(d,t))=Proj(P(d)) represents the pixel coordinates of the device d in the homogeneous world coordinate system, which are mapped to the video frame by the projection mapping function Proj.

[0144] C tech (d) represents the technical capability parameters of device d, which are collected in step S10;

[0145] τ(d) is the response time of device d;

[0146] ρ(t) is the real-time rhythm curve of the video, which is calculated using the rhythm quantization B obtained in step S30 through the following formula:

[0147] ;

[0148] Where P is the beat period, P=60 / B; A is the beat intensity coefficient, an adjustable hyperparameter with a value range of [0.1,0.5], used to control the prominence of the beat; when A=0, there is no beat fluctuation, and the larger A is, the stronger the beat; E(t) is the emotional intensity base value, calculated by the following formula:

[0149] ;

[0150] Among them, t k For the k-th critical time point, 0 = t0 < t1 < ... < t N-1 =T, where T is the total duration; e k For time t k The emotional intensity ranges from [0,1], where 0 represents the lowest level of depression and 1 represents the highest level of excitement.

[0151] Proj(P(d)) converts device world coordinates to video pixel coordinates, establishing a spatial correspondence between the physical device and the video content. Specifically, it converts to:

[0152] ;

[0153] Where, P(d)=(X w ,Y w Z w ,1) T Let d be the homogeneous world 3D coordinates of device d;

[0154] K is the virtual camera intrinsic parameter matrix (3×3), which determines the imaging parameters of the video image, such as focal length and principal point;

[0155] [R|t] is the virtual camera extrinsic parameter matrix (3×4), where R is the rotation matrix used to control the rotation attitude of the camera lens, and t is the translation vector used to control the three-dimensional position of the camera. It defines the position and attitude of the virtual camera in the homogeneous world coordinate system.

[0156] (S80) The independent synchronized audio track generated in step S60, the lighting equipment control instructions generated in step S70, and the timeline and Cue point information determined by the light show script generated in step S40 are aligned and encapsulated on the same timeline with millisecond-level precision to generate a standardized light show performance file that can be played offline and supports direct loading by the on-site controller.

[0157] (S90) Using the standardized light show performance file generated in step S80, the light show is simulated and previewed in a virtual scene constructed using the data collected in step S10 to achieve the final effect in a real environment.

[0158] Specifically, in the virtual scene constructed using the data collected in step S10, the standardized light show performance file generated in step S80 is loaded for high-fidelity simulation operation, and the final effect of the light show in the real environment is previewed from any perspective.

[0159] (S100) Based on the final effect of step S90, the standardized light show performance file generated in step S80 is optimized to form the final version of the performance file and the lighting equipment control instruction set.

[0160] This step aims to optimize the standardized light show performance file generated in step S80 to obtain a better final version of the performance file and lighting equipment control instruction set.

[0161] Specifically, after watching the preview, creators can directly provide feedback using natural language (e.g., "speed up the opening," "warm up the main color scheme," "make the outline lights on the building's roof brighter during the climax"). The large language model, through fine-tuning, analyzes these instructions, transforming them into specific optimization intentions. It intelligently determines the instruction's affiliation and automatically distributes it to the corresponding module in the preceding step: instructions pointing to visual effects (color, style) are distributed to the textural video model in step S60; instructions pointing to motion rhythm are distributed to the constraint weights or rhythm modulation parameters in step S60; and instructions pointing to specific device effects are precisely located to the mapping function in step S60. If the modification involves the core structure of the script, it triggers the regeneration of video clip sequences and corresponding music. After iteration, the final performance file and control instruction set are output, which can be directly used for live-action light show performances.

[0162] Based on the above steps of automatic light show script generation using AIGC semantic understanding, the automatic light show script generation system based on AIGC semantic understanding can output a light show script that avoids the disconnect between virtual lighting effects and physical equipment.

[0163] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of the claims of this application.

Claims

1. A method for automatically generating light show scripts based on AIGC semantic understanding, characterized in that, Includes the following steps: (S10) Collect spatial metadata, equipment metadata, semantic capability metadata, environmental metadata and material basic data. Among them, spatial metadata includes the on-site three-dimensional dimensions of the light show, coordinates inside the building, landscape nodes and installation coordinates of the lighting equipment; equipment metadata includes the lighting equipment model, color gamut, brightness range, response speed and control protocol; semantic capability metadata is the light effects that the equipment can present; environmental metadata includes coordinate system, viewpoint, perspective relationship and camera intrinsic / extrinsic parameters; material basic data includes pre-stored videos, pictures and lighting effect scripts. (S20) Collect users' natural language creative ideas; (S30) The parsing module of the fine-tuned LLM model is used to perform intent recognition, keyword extraction, style classification and sentiment quantification on the user natural language creative collected in step S20, and output a structured demand feature vector V=[S,C,B,T,D], where S is the application scenario, C is the style vector, B is the rhythm quantification, T is the total duration and D is the device and environment information. (S40) Integrate the demand feature vector V generated in step S30 with the device metadata and semantic metadata collected in step S10 into a complete input context and input it into the light show script generation module of the fine-tuned LLM large model to output the light show script. (S50) Perform device hardware verification on the light show script output in step S40, replace the light and shadow effects and color parameters that are not supported by the existing devices in the light show script with equivalent light effects and color parameters of the same style and that can be achieved by the existing devices, and output the corrected light show script. (S60) Using AIGC technology, the corrected light show script output in step S50 is used to generate a corresponding video segment sequence and generate or match the corresponding music, and outputs a video sequence corresponding to the light show script, an independent synchronous audio track aligned with the video sequence at the millisecond level, and a time-emotion rhythm curve generated based on the independent synchronous audio track. (S70) Using the video sequence generated in step S60, the time-emotion rhythm curve, and the digital twin of the light show venue constructed using the data collected in step S10, light equipment lighting effect control instructions are generated. (S80) The independent synchronized audio track generated in step S60, the lighting equipment control instructions generated in step S70, and the timeline and Cue point information determined by the light show script generated in step S40 are aligned and encapsulated on the same timeline with millisecond-level precision to generate a standardized light show performance file that can be played offline and supports direct loading by the on-site controller.

2. The method for automatically generating light show scripts based on AIGC semantic understanding according to claim 1, characterized in that, In step S40, when the fine-tuned LLM model outputs the light show script, the format of the light show script is restricted by the strongly constrained standardized prompts, and a light show script with a fixed four-segment narrative structure is generated according to the following constraint rules: (R1) The narrative structure is forcibly divided into four sections: opening, development, climax, and ending; (R2) Mark each script segment with the time range, applicable lighting area, main lighting effect type, dominant color, movement rhythm, and accompanying audio mood; (R3) All light effect description words used in the light show script can only be selected from the semantic capability metadata collected in step S10.

3. The method for automatically generating light show scripts based on AIGC semantic understanding according to claim 1, characterized in that, In step S60, before outputting the video sequence, the video sequence is constrained and enhanced using the multiphysics decomposition total loss function; wherein, the multiphysics decomposition total loss function is: L total =L temporal +λ p (t)×L multi-physics +λ l ×L lighting +λ c ×L color ; Among them, L temporal The temporal continuity loss is calculated using the following formula: ； In the formula, I t and I t-1 t and (t-1)th frames of video, respectively; H is the height of the video frame; W is the width of the video frame; ‖*‖ is the L2 norm, which characterizes the intensity of pixel differences between frames; L lighting The loss of lighting consistency is calculated using the following formula: ； In the formula, The estimated light source direction for the t-th frame of the video; The estimated light source direction for the (t-1)th frame of the video; The intensity estimated for the t-th frame of the video; The intensity estimated for the (t-1)th frame of the video; L color The loss of color consistency is calculated using the following formula: ； In the formula, N is the number of sampled pixels, and its value is N=(H×W) / 64; p i For sampling pixels; flow(p) i ) represents the video from frame t-1 to frame t in p i Optical flow at a point; λ p (t) represents the dynamic weights of the physical constraints, which are calculated using the following formula: λ p (t)=σ(a·v avg (t)+b·v avr (t)+c); In the formula, v avg (t) represents the average velocity of the optical flow field in the t-th frame of the video; v avr (t) represents the variance of the optical flow field in the t-th frame of the video; a = 2.0, b = 1.0, c = -1.0; σ is the Sigmoid function; λ l λ represents the weight for light and shadow consistency. c This represents the color consistency weight, and is related to λ. p (t) and λ l Satisfying λ l =λ c =0.5×(1-λ p (t)); L multi-physics The multiphysics loss is calculated using the following formula: L multi-physics =w g ×L gravity +w i ×L inertia +w f ×L fliud +w e ×L elastic ; In the formula, L gravity For gravitational field constraints, the following formula is used for calculation: ； Among them, a t is the motion acceleration estimated for the optical flow field; g is the standard gravitational acceleration vector; L inertia For inertial motion constraints, the following formula is used for calculation: ； Among them, v t v is the average motion velocity of the optical flow in the t-th frame of the video; t-1 The average motion velocity of the optical flow in the (t-1)th frame of the video; L fliud Assuming fluid dynamic constraints, the following formula is used for calculation: ； in, To perform Laplacian operations on intensity or grayscale images of video frames; L elastic For elastic deformation constraints, the following formula is used for calculation: ； Where, d t The deformation displacement of the characteristic point; w g w i w f and w e These are the gravitational field weight, inertial field weight, hydrodynamic weight, and elastic field weight, respectively.

4. The method for automatically generating light show scripts based on AIGC semantic understanding according to claim 3, characterized in that, In step S70, the color values ​​of video pixels in the video sequence are processed through projection mapping, capability constraints, response compensation, and rhythm adjustment using a physical space-pixel space joint solution function to generate lighting effect control commands for the lighting equipment; wherein, the physical space-pixel space joint solution function is: Out(d,t)=M(I(u(d,t),v(d,t),t),C tech (d),τ(d),ρ(t)); In the formula, M(*) represents a multi-dimensional mapping; I(u(d,t),v(d,t),t) represents the color value of the video at pixel coordinates u(d,t) and v(d,t) and time t. (u(d,t),v(d,t))=Proj(P(d)) represents the pixel coordinates of the device d in the world coordinate system, which are mapped to the video frame by the projection mapping function Proj. C tech (d) represents the technical capability parameters of equipment d; τ(d) is the response time of device d; ρ(t) is the real-time rhythm curve of the video, which is calculated using the rhythm quantization B obtained in step S30 through the following formula: ； Where P is the beat period, P=60 / B; A is the beat intensity coefficient, with a value range of [0.1,0.5]; E(t) is the base value of emotional intensity, calculated by the following formula: ； Among them, t k For the k-th critical time point, 0 = t0 < t1 < ... < t N-1 =T, where T is the total duration; e k For time t k The emotional intensity, with a value range of [0,1].

5. The method for automatically generating light show scripts based on AIGC semantic understanding according to claim 4, characterized in that, Proj(P(d)) converts the lighting equipment installation coordinates into video pixel coordinates, establishing a spatial correspondence between physical equipment and video content. Specifically, it converts the coordinates into: ； Where, P(d)=(X w ,Y w Z w ,1) T Let d be the homogeneous world 3D coordinates of device d; K is the virtual camera intrinsic parameter matrix (3×3); [R|t] is the virtual camera extrinsic parameter matrix (3×4), where R is the rotation matrix used to control the rotation attitude of the camera lens, and t is the translation vector used to control the three-dimensional position of the camera. It defines the position and attitude of the virtual camera in the homogeneous world coordinate system.

6. The method for automatically generating light show scripts based on AIGC semantic understanding according to claim 1, characterized in that, Also includes: (S90) Using the standardized light show performance file generated in step S80, the light show is simulated and previewed in a virtual scene constructed using the data collected in step S10 to achieve the final effect in a real environment; (S100) Based on the final effect of step S90, the standardized light show performance file generated in step S80 is optimized to form the final version of the performance file and the lighting equipment control instruction set.

7. A system for generating light show scripts using the AIGC-based semantic understanding-based automatic light show script generation method described in claim 1, characterized in that, include: The data acquisition unit is used to collect spatial metadata, device metadata, semantic capability metadata, environmental metadata, and basic material data; The human-computer interaction unit is used to input users' natural language ideas and natural language requirements. The light show script generation unit is used to generate light show scripts based on natural language creative ideas and natural language requirements. The light show script generation unit has a built-in fine-tuned LLM model, which includes a parsing module and a light show script generation module. The parsing module is used to perform intent recognition, keyword extraction, style classification, and sentiment quantification on the user's natural language creative ideas and natural language requirements, and outputs a requirement feature vector V. The light show script generation module is used to generate a light show script using the data collected by the data acquisition unit and the requirement feature vector V output by the parsing module. The AIGC multimodal content generation unit is used to generate an AIGC video sequence, an independent synchronous audio track aligned with the AIGC video sequence in milliseconds, and a time-emotion rhythm curve based on the light show script. The control instruction generation unit is used to generate the final version of the performance file and lighting equipment control instruction set based on the AIGC video sequence, the independent synchronous audio track aligned to the AIGC video sequence in milliseconds, the time-emotion rhythm curve, and the data collected by the data acquisition unit; The data acquisition unit is connected to the light show script generation unit and the control instruction generation unit, respectively. The light show script generation unit is connected to the human-computer interaction and AIGC multimodal content generation units, respectively. The AIGC multimodal content generation unit is connected to the control instruction generation unit.

8. The system according to claim 7, characterized in that, It also includes a constraint enhancement unit for multi-condition constraint processing of AIGC video sequences, an AIGC multimodal content generation unit that is communicatively connected to the constraint enhancement unit, and a constraint enhancement unit that is communicatively connected to the control instruction generation unit.

9. The system according to claim 7, characterized in that, It also includes a central control unit, which is connected to the data acquisition unit, the human-computer interaction unit, the light show script generation unit, the AIGC multimodal content generation unit, and the control command generation unit.

Images